After the English adopted fixed surnames in the Middle Ages, they traditionally only used two names: a first name and a last name.

Colonial America was founded not only by the English, but by people of many European nationalities, including significant populations of German, Dutch, and Scandinavian immigrants.

As explained in the link above, the fixed surname system wasn't the only naming system in use in Europe in the 1600s and 1700s. The Dutch, Scandinavians, some Germans, and others used a patronymic naming system, which gave each person two separate last names. A Dutch person's name back then would be in the format of "Jan Pietersen van der Bilt" which translates "John, Peter's son, of the Hill". It wasn't figurative. This person John's father was literally named Peter, and they lived on the hill. When Jan's kids were born, their first last name would be "Jansen" and their second last name may or may not be "van der Bilt", depending if you still lived on the hill.

On top of that, the Dutch had a strict first name naming custom as well. The two firstborn sons were named after their two grandfathers, and then the third born was named after dad. Same for girls, except with their grandmothers and mom.

Further, if any of these kids died while young, and you were still having more kids, then the next born kid would be the replacement kid, honoring the dead sibling by taking his or her name.

Because of this, it was inevitable to have many generations in a family with repeating names. Jan Petersen's firstborn son will be named Peter. Peter Jansen. And Peter Jansen will name his firstborn son Jan. Jan Petersen. And so on.

So it was quite normal for Jan Petersen, once he was a grandfather, to have multiple grandsons all named Jan. If your sons had been named Peter, Anders, and Jacob, then your three Jan grandkids would be Jan Petersen, Jan Andersen, and Jan Jacobsen, which is how you would differentiate them.

If it sounds complicated, then you might have sympathy for the English government who took over New York in 1664. In 1680, fed up with the "Dutch names" of most of the New York population, the English began forcing the Dutch to adopt fixed last names. In a lot of cases, their second last name became their permanent last name, which is where last names like "Vanderbilt", "Van Buren" ("of Buren", the name of a town), and "Dewitt" ("the white") come from. Other families got to use their first last name such as "Peterson", "Johnson", or "Anderson", though only a handful since the point was to avoid confusion. Other families didn't use either of their old last names, but got a new one entirely, often for now-lost reasons. By the end of the 1600s, nearly all of the Dutch people in New York and New Jersey had made the change over to fixed surnames from the old Dutch system of patronymics.

However, still being Dutch and going to the Dutch church, they continued to adhere to the first name naming custom when baptizing their kids. So if Jan had three sons, who all fathered sons of their own, he would still end up with three grandsons named Jan.

Under patronymics, this wasn't a big deal, because the three Jan grandsons would all have different last names: Jan Petersen, Jan Andersen, and Jan Jacobsen, as explained before.

But now, with fixed surnames, all three kids would have the same last name: John Vanderbilt, John Vanderbilt, and John Vanderbilt.

So to differentiate between the three John Vanderbilts, all living in proximity to each other and all around the same age, the Dutch gave their kids middle names, after their fathers: John Peter Vanderbilt, John Anders Vanderbilt, John Jacob Vanderbilt.

This issue took a couple generations to develop, so it wasn't until the mid-1700s in America before these new-fangled middle names became evident. By the time the American Revolution broke out in 1776, it became quite normal to see soldiers recorded with a middle name, particularly if they lived nearby cousins or other relatives with the same first and last name as they had.

People of English/British heritage sometimes adopted a middle name, too, though with much less frequency. It was mostly confined to the upper classes, where a father wanted to honor both his own father as well as an in-law. So John Adams named his son John Quincy Adams (instead of John Adams, Jr.) because a famous in-law's last name was Quincy. But this was still relatively rare in America amongst the British population in the mid-1700s. Six of the first seven presidents did not have a middle name, nor did most of their Cabinets.

After the Revolution, the Dutch, German, and Scandinavian communities broke up because many of the younger people moved West or went to Canada, and also intermarried with English wives/husbands. Those who remained back East saw their towns overwhelmed by immigrants from different parts of Europe.

Although the original communities disappeared, their "middle name" custom survived. John Jacob Vanderbilt would give all his own kids a middle name, despite the fact he married an Englishwoman and moved West so he no longer attended the Dutch church. Within a couple generations, the original reason for giving American children a middle name was lost, but the middle name custom remained, and began to spread. Middle names were ever more frequently chosen for aesthetic purposes rather than in honor of a relative.

By 1900, having a middle name was no longer seen as an ethnic Dutch/German/Scandinavian custom, but an American one. So even immigrants to the U.S. who did not come from northern Europe would often give their American-born children a middle name in an effort to assimilate.

During this same time frame, 1700-1900, in the UK, however, none of this happened. So people born in the British Isles continued to have only a first name and a last name.

It has only been in the last half century or so that British children have been given a middle name, an influence of the American custom.

TL;DR: It's a leftover of Dutch, German, and Scandinavian patronymics in colonial America, which the English/British government sought to eliminate in the late 1600s and early 1700s, but it resulted in the American custom of middle names. Since World War II, the British have more widely adopted this custom due to American influence.