People tend to think that the European royal houses were long, unbroken lines stretching back into the mists of time, aka "the Middle Ages". In fact, that's not the case at all, and intermarriage between royal houses was rarer -- and was not repeated, close cousin marriage -- until only a few hundred years ago (roughly the 17th century). Thus, it is only recently that intermarriage has been prevalent enough to have obvious adverse effects.

First, the factors that prevented this in the Middles Ages. The catholic church was relatively proactive in preventing marriage between closely related people. Your relative relationship to other people is called consanguinity. In the ninth century, the church required seven degrees of separation between bride and groom, and then, when that turned out to be impractical for several reasons, they reduced it back to four degrees. This means that the couple cannot share a great grandparent. There also was a lot more disruption in royal and noble lines than there would be later, with far less emphasis on royals marrying royals and more on marriage alliances based on building power and wealth. These factors helped mitigate the effects of inbreeding.

This isn't to say that royals did not sometimes marry cousins, it was more common in Spain because the houses of Castille and Aragon kept intermarrying. It just was usually not as overt as literal first cousins marrying, and the family trees in the Middle Ages weren't as tangled as they would become in the early modern period.

What changed was that, in the early modern period (in the sixteenth and seventeenth centuries) several different families, but especially the Hapsburgs, began using marriage alliances as a strategic way to consolidate power, and were more overt and successful than previous royal families. Simultaneously, Europe divided itself in half based on religion, which limited choices of marriage partners for protestant and catholic monarchs. The divine right of kings and the heavy emphasis of royalty and nobility as a matter of God-given superiority also became much more emphasized, which pushed monarchs into purposefully seeking out royal marriages, rather than just advantageous marriages. There were also arguably two sub-factors, first an explosion of eligible royal houses and second the split of the Hapsburgs between Austria and Spain (which meant that two catholic, royal houses with strong incentive to marry into each other were descended from the same person).

This is when royal family trees start to look more like family spiderwebs. Carolos/Charles II of Spain (1661-1700) is one of the first examples of a monarch who probably suffered physical and mental problems because of serious inbreeding (although we can't say for certain without the ability to do genetic testing). There were also major political problems for Spain when he died without and heir, leading to the War of Spanish Succession.

Now, did "royalty" ever figured out that inbreeding was bad? Well, certainly modern royalty is aware of that fact. We also know that the late nineteenth century royal families were painfully aware of the fact that Queen Victoria's children were the carriers of hemophilia, even if they were unaware of the exact genetic causes for this. It's possible that earlier royals were also aware of this. However, royalty continued seeking to marry royalty until the twentieth century, and then they still attempted to marry into the nobility until the very late twentieth/early twenty-first century.