The Swedish royal line ended in 1818. King Gustav IV Adolf was deposed in 1809, following Sweden's loss of Finland following the Finnish War. Gustav IV Adolf was replaced by his uncle, Charles XIII. However, seeing as Charles was childless, parliment elected Prince Christian August of Augustenborg - a Danish prince - as his heir. (Despite the fact that Gustav Adolf was still alive, as was Gustav Adolf's under-aged son - having been deposed, he was not asked to return to the throne).

However, Prince Christian August died in 1810. This left the decrepit, aging Charles XIII again without an heir - so a Swedish baron named Karl Otto Mörner offered the position of heir-presumtive to Jean Baptiste Bernadotte, the Marshal of France. Bernadotte had no familial ties to the Swedish royal family, but his election won favour with the parliment, and Bernadotte became king once Charles XIII died in 1818 - adopting the name Charles XIV.

... And that's the very condensed version of how a French officer became the king of Sweden.

There have been a few famous instances where a royal family didn't produce an heir. Looking at English history we have Queen Elizabeth 1 who did not have any children.When she died that was the end of the Tudor monarchs of England. The throne passed on to James Stewart who had a claim to the throne from his great-grandmother.

Another kind of example is Charles the Bold, the Duke of Burgandy. He died in battle leaving unmarried 19 year old daughter as his heir. The duchy of Burgandy was one of the larger duchies that existed in Europe, consisting of a large chunk of the low countries and part of modern day France. After Charles died his holdings were essentially split between France and the Hapsburg's.

By way of an introductory example, the Portuguese Succession Crisis of 1580 came about when Sebastian I died in battle at the age of 24 and without heirs. Ultimately that led to the King of Spain, Philip II, taking the throne instead. It only gained independence again in 1640.

An example I can give more detail on is the War of the Spanish Succession from 1701 to 1714. Carlos II 'the sufferer', the now deceased King of Spain, was highly inbred and totally unable to produce an heir, so the issue of who would take over after his death had dominated European politician's thoughts for some time. Key among the rival contenders were the Austrian Hapsburgs (Carlos II having been a Hapsburg himself: the extended family wasn't necessarily wiped out but the immediate succession certainly was) and the French Bourbons. Louis XIV of France was an exceptionally powerful player at the time.

So they came up with different solutions. The French and Austrians developed a partition treaty that would have split up various Spanish territories between them in exchange for the Hapsburgs getting the monarchy, but this was complicated by Carlos II himself, who named Louis XIV's grandson, Philip, as his heir with a couple of conditions: no unified monarchies of France and Spain, and no splitting up of the Spanish Empire. Louis XIV was presumably quite surprised to receive this offer when Carlos II died and decided quickly to accept it and waged the aforementioned war. It was a massive conflict that drew in most of the major powers in Western Europe and ended with Philip, Louis' grandson, on the throne of Spain but with reduced Spanish territories: the Netherlands, for example, were won by the Austrians, while Britain gained the exceptionally profitable Asiento, a trading monopoly with the new world.

So the answer to your question is: yes some royal families failed to produce an heir, even significant ones. Their land holdings could be absorbed or have wars fought over them, depending on the situation. The kingdom could indeed be lost, as with Portugal, but that didn't mean it couldn't ever be up for grabs again.