The key thing to remember is that the Roman Empire in all of its nearly 1500 year history (from Augustus to the Fall of Constantinople) never developed rules of succession like those of European dynasties in the central and later middle ages and the early modern era.

Roman law offered a fairly loose definition of family as those under the authority of a paterfamilias (patriarchal authority figure), which often placed adopted children on the same footing as biological children. When it came to the inheritance of property, while the general expectation was that the paterfamilias would make provisions for everyone in the family, including his slaves and clients, through the use of written wills he could designate who inherited what and disinherit children as he saw fit. The empire was often treated in the same way i.e. the so-called five good emperors who reigned from 96 - 180 AD (Nerva, Trajan, Hadrian, Antonius and Marcus Aurelius) succeeded each other through being the adopted sons of the previous emperors - they were not biologically related to each other. For the "proper" dynasties, things were little different. For example for the so-called Julio-Claudian dynasty (the five emperors that everyone's heard of - Augustus, Tiberius, Caligula, Claudius and Nero), Tiberius succeeded his step-father and adoptive father Augustus. Tiberius was then succeeded by Caligula, the grandson of Augustus through his daughter Agrippina the elder and the son of Tiberius' adopted grandson Germanicus. After Caligula was assassinated the senate allowed Claudius to take power and Claudius married Caligula's sister, Agrippina the Younger, and adopted her son Nero as his successor. Thus while all five emperors were related to each other by common ancestry or marriage, none of them directly succeeded their biological fathers or grandfathers. Many later dynasties carried that on as well i.e. for the Justinianic dynasty of Roman emperors, Justin I was succeeded in 527 by his nephew and adopted son Justinian who was then succeeded by his nephew and adopted son Justin II in 565 - the last two emperors of that dynasty, Tiberius II and Maurice, weren't even biologically related to each other or the preceding three emperors, being members of it through adoption by the previous ruler. Essentially we have to wait until the late 11th century for the House of Comnenus to find a Roman imperial dynasty remotely comparable to those of medieval and early modern Europe, and though they managed unbroken father-to-son succession for four generations (nothing like the Capetians, who managed it for more than 300 years, yet an unparalleled achievement in Roman history up until that point all the same), after a century in power they descended into brutal dynastic infighting with their collaterals and in-laws.

The other two things to factor in are the legacy of the Republic and the importance of the Roman army as powerbrokers. While its true that most of the continuity the emperors of the principate period (27 BC - 235 AD) claimed with the Republican constitution was largely just propaganda to get a people whose identity had been based on a rejection of monarchy for almost 500 years to accept it, in some ways the Republic still exercised a meaningful political legacy at least as late as the 12th century. For starters, constitutional theory throughout the imperial period held that emperors were meant to be popularly acclaimed by the senate, army and people of Rome (later Constantinople) and in some cases they actually took the initiative to elect them i.e. when Emperor Michael V was overthrown by a popular revolt in 1042, the sisters Zoe and Theodora were elected co-empresses by the senate and people of Constantinople. More importantly, however, it gave an enduring sense of an abstract impersonal state - that the Roman res publica (what the Romans called their state/ political community throughout their history) / politeia (in Greek) was a self-sustaining entity, not grounded in loyalties to any particular dynasty. This is the other great contrast to, say, Capetian France or the early modern Hapsburg realms.

Turning to the role of the army, the transition from Republic to Empire essentially began in the 1st century BC because of the power bases independent of the senate and other traditional republican offices and institutions that the generals had acquired from the Marian reforms. The Roman Empire was in certain ways a military junta from the outset, though it took until the crisis of the 3rd century for the mask to properly slip - after the assassination of Emperor Alexander Severus by the praetorian guard in 235, 50 emperors were proclaimed by the Roman military in the next 49 years. The emperors responsible for the Roman Empire's recovery from the crisis - Aurelian, Diocletian, Constantine the Great etc - were all soldier emperors who gained and consolidated power through the Roman army. Though the "military anarchy" of the 3rd century was quickly left in the past, military coups remained an important part of imperial Roman political life thereafter - the last time the Roman army tried to topple an imperial dynasty was in 1295 when the general Alexius Philanthropenos tried to usurp the throne from the unpopular emperor Andronicus II Palaiologus.

In short, Roman dynasties were so short-lived because there were no hard and fast rules of succession to keep the imperial title within the family and, while the Republic was gone, state and dynasty were not synonymous with Romans being loyal to the Roman state first and to the reigning dynasty second, and there was always a strong element of military junta in the backdrop.