Short answer: they didn't. King et al. cites a "false-paternity rate of ~1–2%" (1, 2, 3). Genealogists calculate this by looking at special kinds of DNA. Men inherit the Y-chromosome only from their fathers since women don't have a Y-chromosome. So if you have two putative first cousins, they should have nearly identical Y-chromosomes that they both inherited from their common grandfather. But if their Y-chromosomes differ greatly, then a so-called "false paternity event" occurred and their biological father/grandfather is actually someone else. The same idea works with mitochondrial DNA, which is only inherited from mothers, in the case of the biological mother/grandmother being someone else. Genealogists have extended this same idea to more distant cousins by sequencing many members of large families with the same surname to arrive at the 1% to 2% estimate.

Your question was about royalty specifically, though. Fortunately, there is a fascinating case involving Richard III, the last King of England in the Plantagenet dynasty, famously depicted as a hunchback by Shakespeare. Richard was killed during the Battle of Bosworth Field ("A horse, a horse, my kingdom for a horse!") but his grave was lost over the last half-millennium. In 2012, some archaeologists found a skeleton that might be Richard, but they couldn't be sure. So King et al. sequenced DNA from the skeleton, and from a number of putative biological relatives down the male/female line who should have nearly identical Y-chromosome/mitochondrial DNA, barring adoption or a false paternity/maternity event.

On the female line, the skeleton had nearly identical mitochondrial DNA as two living people whose family trees claimed they were 19 and 21 generations removed from Richard III. It is usually much more difficult to find female line descendants since women traditionally changed their surnames upon marriage, but the genealogists managed to do it. Thus, it is virtually certain that the biological mother of the skeleton is the same person as the biological 19th great-grandmother and 21st great-grandmother two putative descendants of Richard III's mother. Therefore, the skeleton is almost certainly Richard III.

The male line was more complicated. The skeleton had vastly different Y-chromosome DNA as five living people whose family trees claimed they were 24 to 26 generations removed from Edward III, Richard's great-great grandfather. Thus there must have been a false paternity event somewhere, either between Richard and Edward III, or between Edward III and his living relatives.

Source: King, Turi E., Gloria Gonzalez Fortes, Patricia Balaresque, Mark G. Thomas, David Balding, Pierpaolo Maisano Delser, Rita Neumann, et al. 2014. “Identification of the Remains of King Richard III.” Nature Communications 5 (1). Springer Nature. https://doi.org/10.1038/ncomms6631.