It seems - and correct me if I'm wrong - to be a uniquely US thing, at least for non title holders. Kings and dukes etc in Europe are given regnal numbers, but usually in relation to their first names. It's Henry IV, V and VI not Henry Lancaster, Henry Lancaster Jr and Henry Lancaster III.
As a side note, I don't recall ever seeing an example of this tradition being extended after "Name III" - are there any notable examples of "Name IV" etc?
This isn't a particularly American thing. It's actually an older European thing, and other places have their own version of it too.
You might actually get a better answer at /r/genealogy, but this goes back to traditional naming patters in Europe, among both places that used "patronymic" surnames as well as those that used "fixed surnames". For a breakdown of the patronymic customs, a good source is the article "Dutch Systems in Family Naming in New York and New Jersey" by Rosalie Fellows Bailey published in the March and December 1953 issues of the National Genealogical Society Quarterly. For an analysis of traditional English and British naming customs, seek out the book Names and Naming Patterns in England, 1538-1700 by Scott Smith-Bannister. Those are some more scholarly sources, but alternately, you can see a blurb about the same topic in Tracing Your Edinburgh Ancestors: A Guide for Family and Local Historians, but more to the point is this article from Family Tree Magazine.
The short of it is, there used to be a custom in Europe in which to name your sons:
And your daughters, too:
These weren't strictly followed in every family, and the custom tended to be stricter in patronymic societies than in non-patronymic. There were customs for fourth-born, etc., children as well (oldest uncles and aunts), though these were even less strictly followed. The order could also be switched around, too, particularly if one of these relatives were dead and the others were not. So if dad's dad was still alive but mom's dad was dead, then mom's dad would be the name of the first-born son.
Further, since a lot of children died during childhood back in the old days, for a long time, it was quite common for the next born child of that gender to be the "replacement" child, named in honor of the deceased child.
The point being, given that everybody had as many kids as they could up until the mid-19th Century, it was quite common for a father to have a son with his same name--his third-born son. Or sometimes a later son if the third-born son died while young.
This also often reduced the pool of names used by a given family, so the first-born child could have that name instead.
For example: John's father is named Peter. John's father-in-law is named William.
John names his first three sons Peter, William, and John. Those are the names of the kids' paternal grandfather, maternal grandfather, and father.
But now his third-born son, John, has a child. Not only is that child's father named John, but so is hid paternal grandfather. So now the first-born son is named John.
And that John has both a father and paternal grandfather named John. So his first-born so is also going to be named John. And on and on.
For a lot of history, there often weren't more than two generations in adulthood at a time, so John and his son were called John Sr. and John Jr. If the third John got old enough before the grandfather died, then he would be John III. But a lot of times, John Sr. died, and unless that John was particularly notable, then John Jr. "became" John Sr. and the third generation "became" John Jr. Nobody was really keeping track how many generations of Johns the family had in its past, just the ones that were alive.
Which is probably why you don't notice familes often going beyond III, because it was rare for three generations to be alive at one time.
One interesting little tidbit about this traditional naming custom was that, sometimes, the paternal and maternal grandfathers had the same name. So John's father is Peter and his father-in-law is also Peter. John's first son is named Peter. But in order not to slight the father-in-law, the second son is given his father-in-law's last name as a first name. So John Smith's first son is Peter Smith, but his second son is Cartwright Smith, named after his father-in-law Peter Cartwright.
A lot of times back before the 1800s especially, if you see an odd first name of a man, check to see what his mother's maiden name was. Often, that's where he got that odd first name. He was his parents' second son, he had two grandfather's with the same first name, so he was given his maternal grandfather's (and mother's maiden) last name as his own first name.
This is the case with Gouverneur Morris, signer of the U.S. Constitution. He was the second-born son to his parents, and his mother's maiden name was Sarah Gouverneur. Same deal with early New York governor DeWitt Clinton. His mother's maiden name was Mary DeWitt.
One family that continued the practice longer than most is the Harrison Family of Virginia that spawned Presidents William Henry Harrison and his grandson Benjamin Harrison. Some of their ancestry is outlined in this old book, though those ancestry books can be difficult to read if you're not familiar. For a better visualization, click "Show" on the family tree on their wikipedia entry. Though it's incomplete, it shows you that there was one branch of the family that used "Benjamin Harrison" for seven consecutive generations. But funnily enough, the President Benjamin Harrison was not in that direct lineage. He only got the name because an uncle had that name, while he had two older brothers named for their two grandfathers (William Henry Harrison and Archibald Irwin). His father then elected to name one of Benjamin's younger brothers after himself (John Scott Harrison, Jr.).
Anyway, you certainly see this in colonial America and into the 19th Century. You occasionally still see it today, though it's not nearly as common, and rarely has it been going on continuously since colonial times.
But this wasn't an American tradition alone. A fully mapped out family tree of English, or Scottish, or Scandinavian, or Dutch, or German families will very often show similar naming patterns. And the use of "Sr." and "Jr." depends on what branch you're looking at: One might have a John followed by Peter followed by John followed by Peter for generations, due to where each father fell in the birth order. None of them have a need for "Sr." or "Jr." But in another branch, you'll see John followed by Peter, followed by Peter, followed by Peter, followed by Peter, where you need a "Sr.", "Jr.", "III", etc., after every name.
In general, though, the actual regnal numbers have only been assigned long after-the-fact, by people studying the family. Aside from the rare "III" when three generations were all alive and living in the same town at one time, most everything beyond a "Jr." wasn't ever really used in the name of the person in their lifetime. John Smith was just known as "John Smith," only bothering with a "Jr." designation if he was likely to be confused by the tax assessor or probate court with his father John Smith Sr.
I hope that was clear enough. You might get a better explanation if you ask over at /r/genealogy, but suffice it to say that this is the result of an old European custom widely, but not universally, practiced. It's not unique to America, but it was brought over and continued during colonial times and into the 1800s. It's much rarer for it to be followed nowadays, and people who do probably aren't following any wider practice other than they want to honor a family member they may know or have known personally.