The rise of large estates in the later Empire was one main culprit which sucked out personnel out of reach from the state to private enterprises.

This was not one huge change but a culmination of multiple factors, stemming from the numerous civil wars, decay of state control, environmental changes, and economic changes which all pressured the 'little guy' the most.

These plebeians became, over time, coloni; or as can be translated, 'proto-serfs' to rich landowners who the state turned to for more and more support but with dwindling control. Taxes changed; adaeratio - commuting taxes from goods to simple cash - became dominant, and thus land - especially in the more rural western part of the empire - became the most profitable good. Landowners and their villas produced goods for the local market or for the overall imperial economy (if they were so fortunate to be close to a safe river or road, with the latter becoming rarer and rarer as time went by). This could be compared to Feudal Manors or Plantations.

Slaves decreased in importance, while they were still around - Melania, a rich lady, freed 8,000 of them in a single act in the later western empire, for example - there was a near constant supply of Roman citizens, a body which exploded in number due to, again tax reforms where all free men and free women were granted citizenship under Caracalla in the early 200s - gave landowners a pool of labour that flocked to them, to work their land for their immediate benefit with the landowner, of course, taking a share for themselves. (slaves were accepted for service in 406, however!)

Another tax was formed, the annona, around the same time of Caracalla, which demanded a fixed portion of produce/agricultural goods which Diocletian expanded upon, which further cemented the role of the landowners.

After all, if the state needs the annona and adaeratio, then the landowners could point out that they needed men to work the land to provide for the tax; soldiers could be found beyond the border or in the city. This does not mean that the Roman military became totally barbarian nor worthless, but moribund in a sense that a cycle had formed where manpower was needed to support the starved apparatus of the state so that apparatus can support other manpower to defend the apparatus, the army dwindled to tens of thousands at most (with untold thousands in private armies/security forces for the landlords - bucellari, because of course the landowners could appease the state by having some standing force to augment the border...and defend their assets as a nice bonus). Meanwhile, the use of barbarians in the army was already a long held tradition since Nero and Marcus Aurelius; and the auxilia and socii before that. When freemen, freedmen and slaves were lacking, Romans already had a precedent to turn to barbarians for tributes and recruits.

By 332 Constantine wrote a law that all coloni who fled their landlords were to be returned to their landlords in chains, by 400 they had become 'servi terrae'; and the fall of plebeian and patrician to humiliores and honestiores, the system became ingrained over the 3rd and 4th century; so much that the 5th was a sort of domino setup where all the cards of those chaotic two prior centuries finally fell in order.

Sources:

The End Of Ancient Rome, ISBN 0-7377-0371-7 (Pages 119-131 particularly)

The Dark Ages, 476-918, 1901 edition, (page 306)

A Companion to the Roman Army ISBN 978-1-4443-9376-7(page 435)