The very short version is that the Roman Empire, and especially after the division of the Roman Empire became de facto formal after 395 AD, the (Western) Roman Empire simply lacked the resources to keep them out - and for good measure the Eastern Roman emperors occasionally sent Germanic peoples threatening them off towards the west (most notably the Ostrogoths).

Also, the numbers involved (especially those moving into Anatolia and Tunisia) were quite limited, and in fact those having moved to Anatolia were easily whiped out by the locals in a (series of) purge(s). The only areas where they were truly able to dominate and spread in great numbers were the UK, the Alpine regions and the Low Countries (something very evident in the languages spoken to this day!). Indeed, Gaul, Spain, North-Africa these hoardes were so limited in number that they were mostly absorbed and all but assimilated into the local resident population (though as the dominating elite).

The long version:

In many ways this disintegration was a proces that started already in the third century (during the infamous crisis), after which the Romans increasingly came to rely on Germanic warriors for recruitment for their armies (the so-called foederati) - though from what I have been able to trace, the explosion of their numbers dates to the second half of the fourth century.

Other outside factors in play were an increased threats on multiple borders: Germanic tribes 'merging' into ever greater and more powerful confederations in direct response to the threat the Romans posed to them, and the replacement of the Parthian rulers with the Sassanid dynasty, a far more aggressive and it seems capable enemy.

This - in combination with the appearance of a plague resulted in economic troubles and a devastating cycle of instability within the empire, economic trouble (inflation), outside raiding/invasion and so on throughout the third century (though not everywhere in the Empire!).

Aurelian, and later on Diocletian and Constantine managed to temporary halt these processes, and stabilize the situation (and to some extent recover the situation).

Mind you, by that time (the early fourth century), at least in the Northwest (the Low Countries), the Romans had been forced to accept some tribes of the Franks living within the Empire, while having their cousins living just across the Rhine.

By the middle of the fourth century a devastating civil war between Constantius and Magnentius weakened the Roman army even further, and arguably Constantius even encouraged Germanic attacks on the part of the Empire controlled by Magnentius. A series of costly battles (and crushing defeats), most notably Mursa, Julian's disastrous campaign into Persia and Adrianople all but whiped out an effective Roman field army, and - at a minimum - forced the Romans to accept a massive Germanic settlement within Roman territory; effectively they had all lost control over a massive part of their border. Worse - by now much of their army consisted of Germanic recruits, especially in the West (though Theodosius had to use massive numbers of them as well when marching against Eugenius at Frigidus).

The Eastern half of the Roman Empire was in many ways extremely lucky, for a number of reasons:

1. Their primary enemy (the Sassanid Empire) had its own problems (domestically and on the Eastern borders), and the division of Armenia (and later on Georgia) between them diffused a flash point -> the threat posed by them disappeared for pretty much a whole century. So no major threat on the Eastern Front, or Southern Front (Africa).

2. The economic heartland of the Eastern Roman Empire (Egypt and Syria) was well protected from foreign invasion. Its most dangerous threat (the Sassanids) was neutralized, and it was out of reach (and easily defended) from Germanic threats by the navy and the walls of Constantinople itself.

3. The Eastern Roman Empire possessed the economic resources to (a) maintain the army needed, and (b) bribe potential enemies to go elsewhere (including paying tribute).

4. The Eastern Roman Empire was able to recruit sufficient troops for its military requirements from within its own borders, and did not have to rely on Germanic troops (in fact, there were a number of purges which eliminated Germanic troops as well as high ranking Germans (generals) within the army.

5. The Eastern Roman dynasty was very stable (luck of the draw to some extent), and the family (together with its circle of advisors) was able to dominate domestic politics, so no dynastic struggles happened.

6. A bit more speculative, but the it seems that the Eastern Roman political elite (including the imperial family) was more pragmatic, far more politically astute and had a much better grasp of the political, economical and military realities than their idiotic equivalents in the West (who made some increadibly stupid decisions).

In contrast to the Eastern half of the Empire, the West had much worse luck:

1. The primary threat to them never disappeared, in fact the treats to them only increased: the Germanic tribes became more powerful, and there was an increasing rivalry (and enmity) with the Court in Constantinople.

2. The economic heartland (Gaul, Britain) of the Western Roman Empire was very exposed to outside invasion. Worse, even when at full strength, economically the Western Roman Empire was much inferior to its Eastern counterpart. Direct result - it lacked the resources to maintain a powerful enough army, or to bribe the enemy in going elsewhere.

3. The military burden - even in the fourth century - placed upon the Western Empire was extremely heavy; not only because of Germanic invasions, but because Western troops were time after time sent east to fight the Persians, or in civil wars (generally Western troops were considered (far) better quality than Eastern troops. From the 350's onward, the Western army never had a period of recovery to have it rebuilt as it was either continually involved in warfare itself (against Germanic invaders or in civil wars) or its troops were sent East. By the early 400's it for all intent and purposes had ceased to exist as an effective fighting force capable of defending its frontiers.

4. The political elite (and one has to include the clergy in there due to the political power of the bishops) seems to have consisted mostly of buffoons (or only slightly better) - which considered petty rivalries and ethnic and religious factionalism more important than actually having capable people running the empire. A prime example of this is the murder of Stilicho by the local Roman elite, which meant that by far the most capable politician and army commander the West had. Worse, it initiated a revolt by troops loyal to Stilicho - and triggered the invasion of Italy by Alaric. They repeated this stupidity again with the murder or Aëtius.

This cycle of invasion, loss of control of territory, loss of income, lack of resources to maintain an army to defend territory (let alone recover lost territory) sent the Western Empire in a downward spiral; over time the resident locals realised they were better of looking after themselves (or were told to look after themselves as allegedly happened to Britain) and/or accepting the overlordship of local elites (the 'Empire of Syagrius) or invading tribes (the Frankish and Visigothic Kingdoms) for their protection.

This process eventually ended with the complete complete collapse in the 3rd quarter of the 5th century. One could argue - however - that the Western Empire lost control over its own destiny with the Germanic crossing of the Rhine (December 31st 405 (Kulikowski) or 406 (according to others)), and sealing its fate with the murder of Stilicho in 408.

Literature:

AHM Jones, The Later Roman Empire (old but still the basis for modern scholarship of the period).

P. Heather, The fall of the Roman Empire and Goths and Romans AD 332-489.

P. Richardot, La fin de l'armée romaine

S. Mitchell, A history of the Later Roman Empire

Matthews, The Empire of Ammianus

Daryaee, The rise and fall of the Sasanian Empire

Hoffman, Das spätrömische Bewegungsheer for some info about the development of the Late Roman Army.

There is much more (but for that I have to dig into my BA thesis, MPhil thesis and some other work from my university days).