How do royal families become royal in the first place?

There are many different ways. Sometimes (as happened in Greece, Romania, Iraq, Jordan, etc.) the ruling powers around the Kingdom agree on who the new royals should be (this was especially common between the Napoleonic Wars and World War II--it was supposed to happen in the Baltics as well, after WWI). Sometimes the parliament/diet/nobility will invite a new royal family in (as happened in the 1688 Glorious Revolution in England). Sometimes a more minor noble's lands grow to the point that he declares himself king or emperor and everyone else sort of accepts it (that's how Egypt ruler of Egypt raised itself to the level of king from Khedive/Sultan in the early 20th century; same things happened in Serbia in 1882, from prince to king). Sometimes a royal family conquers another kingdom or empire, and inherits their titles (that's how the Ottomans became Emperors, for example--they conquered Byzantium). Sometimes they just are victorious enough in politics battle that they become ruler, and thereafter King (Zog in Albania, Serbia in the 19th century through a more circuitous route). Sometimes a king dies without heir, and the the kingdom passes to a cadet line of the same house. Sometimes a similar thing happens, but through marriages, so the kingdom passes to a new house. Often, it is some combination of several of these factors.

For instance, William the Conqueror was Duke of Normandy in France, and his house was descended from Viking raiders who had established themselves in the region (Norman is related to "North man"/Norse). His first cousin once removed (Edward the Confessor) died childless on the throne of England. William was the grandson of Edward's maternal uncle, Richard II, Duke of Normandy. Many potentially claimants, with equally close family lines, but Edward may have chosen William as his official successor. Whatever the legal legitimacy of it, it was favored by some English nobles (possibly including the Archbishop of Canterbury and the Bishop of Winchester) and opposed by others (especially the the Earl of Wessex). In the end, it was military victory (where William was helped by his continental allies, and the fact that two other claimants had fought each other only days earlier) that assured William the English throne. So his rise to the English throne was a combination of legitimacy through family ties, local support by some of the nobility, and military victory. Many other ruling houses established themselves through similar mixes of issues: often, ruling houses would gain a claim legitimate rule through family, and then this claim would have to be enforced through martial means, then the house's legitimacy would be recognized by nobles inside and outside of the state.