When Pompey first invaded Palestine (63 BCE), Herod's father, Antipater, was an advisor to the Hasmonean king, Hyrcanus. Antipater supported the Romans and sufficiently ingratiated himself to Pompey that he was made a procurator of Judea. After Pompey was killed, Antipater supported Julius Caesar and brought troops to rescue him during the Siege of Alexandria (47 BCE). Julius Caesar made Antipater an ethnarch of Judea as a reward, meaning he was allowed to be an independent ruler and didn't have to pay taxes. Herod was made the Governor of Galilee before his father was assassinated in 43 BCE. Antipater and his son, Herod, both became friends with Mark Antony during this time. After Julius Caesar was assassinated, Antipater was initially forced by Cassius to support him against Mark Antony, but after Antipater himself was assassinated, Herod switched loyalties to Mark Antony and Antony made him Ethnarch of Judea. After Antony lost the war to Augustus. Herod, Herod went to Augustus, admitted he had been loyal to Antony and promised he would be just as loyal to Augustus. Augustus was impressed enough allowed him to remain in his position and made him a client king.

Th Romans seem to have liked him because he was an effective administrator in addition to being a good ass-kisser.

The reason they didn't nominate anyone from the Hasmonean family was because there weren't any left. Herod made sure of that himself by killing the last one (Hyrcanus) before going to give his speech to Augustus.

Even before that, the Romans had recognized that Hyrcanus was a weak leader and that Antipater was the real power and brains who was pulling Hyrcanus' strings.

The source for virtually all of this is Josephus in both Wars (books 1 and 2) and Antiquities 17.

The TLDR is that first Antipater and then Herod were both good at sucking up to the Romans and were both recognized by the Romans as being smart and effective adminstrators.