To put it briefly, disease leading to a decreased population, economic distress, rebellions, civil wars, misleading numbers of Roman troops, and the actual size of the empire itself are all theories for Rome’s decline and difficulty in fighting outside threats. Due to the size and complexity of the answer required to fully explain this question, I would like to focus on the military aspect.
By the time of Augustus, Roman expansion of territory exponentially decreased. As a result, Roman troops and legions became increasingly tied to the frontier areas. By the 2nd century A.D., temporary frontier forts built by the legions to provide shelter in winter became permanent settlements representative of small cities. As a result, troops became increasingly bound to one particular region of the empire. By the 4th century A.D., new reforms took place trying to resolve this problem, splitting the army into 2 major segments, the limitanei and a strategic reserve called the comitantenses. The limitnanei’s main job was protecting the frontier borders from minor raids by barbarians, who numbered only a few hundred in most instances. Unlike the limitanei, the comintansenses were designated as the mobile reserve, and designated to move to any threatened area and remove the danger. Unfortunately, the most important part of this reform, the comitansenses, failed in many respects, mainly the comitansenses also became tied to a certain region or province, negating the purpose it was created. Having the Roman troops isolated and tied to specific areas in the empire caused great harm in many respects like civil war.
Roman troop’s identity became connected to the local regions and its commanders. When one emperor died or was seen as illegitimate, a commander with a significant army could declare himself emperor. In other instances, the local military forces themselves would declare a new emperor. This caused huge succession problems and led to an extremely dangerous pattern of civil war. One saw many examples of this in the 3rd and 4rth century, the most famous example being Constantine whose Gaul based army militarily defeated the armies of the East in order to proclaim him emperor. The civil wars played a huge part in weakening the Roman military, making it less effective in defending the empire from outside invasions.
Economic distress, changes, and population decrease also weakened the Roman military. For a large part of their history, the legions built and supplied their own weaponry and armor from military foundries. Essentially, if there was a need for equipment, the legion could supply that need themselves. They were effectively self-sufficient. By the late Roman Empire, weapons and armor manufacturing had been given to nearby cities and professionals who built the equipment and supplied the local Roman troops. This could be a huge liability especially when disease, war, or political corruption was in play. Consequently, local Roman troops could suffer from lack of supplies not representative of the legions from older eras. Local political corruption is key here and leads me to my next point, the actual size of the Roman army and its potential for being a paper tiger.
It is important to preface this and further arguments with the fact that the late Roman Empire’s military was still the most effective, disciplined, and best equipped forces in the Western World, yet it is also important to note several weaknesses that had developed, for instance, its actual size compared to its paper size. Some scholars estimate the Late Roman Empire fielded close to 700,000 troops! This number is nearly twice the size of the Augustus era troop level of around 400,000. However, many of these troop numbers were most certainly faked. It was common to see a legion on paper fielding thousands of men, but in reality, its real numbers were much smaller. One reason for this was the army’s behavior at the time. Due to the need for field mobile armies to fight off multiple threats on the borders, quick reaction forces would be scratched together from multiple frontier garrisons and reserves. These forces were called vexillations. Eventually over time, these vexillations became defacto legions. As a result, one saw a drastic explosion in the number of legions, but saw little quantitative increase in manpower. Essentially, the legions split into even smaller legions. Another reason for inaccurate troop numbers was political corruption. This cannot be emphasized enough. In order to receive more money for their coffers, local officials would inflate the number of troops in their vicinity, leading to the false conclusions on the number of military men. A third reason for poor troop count is the fact of military attrition itself. Men would be killed in border skirmishes or minor campaigns while others would die every year from disease. Unlike older days where it was easier to recruit, the Roman Empire suffered from many plagues, diminishing the population and making it difficult to replenish the military. These three factors help explain the reason for lower than expected numbers of Roman troops.
Although a minor change, one could point to the decreasing quality of the Roman army. Some of the evidence on the lack of quality of the Roman military was its lack of discipline in battle compared to its earlier days. In the time of Julius Caesar and Augustus, the legions would advance slowly and silently over the field and in the final few meters charge and yell. This action took an extremely large amount of discipline and training. In the final decades of the Western Roman Empire, the troops did the opposite; charging and howling similarly to the barbarians they were fighting. One of the biggest factors for this change in tactic was the inevitable influence of incorporating larger numbers of barbarians into the military. It is also important to note the Roman troops had been configured and used to fighting small border clashes. As a result, Roman vexillations or legions had rarely fought together in pitch battle or even had the opportunity to train together. This decrease in the quality of the Roman troops leads me to my final point.
The Roman Army was partly incapable of fighting off the barbarian invasions due to its unwillingness to fight in open battle. This was common idea in ancient warfare. Much of the campaigning would be hallmarked mainly by maneuver and skirmishing with few battles. In many instances, only one large battle would be fought. Add this to the difficulty the Roman army faced in the 3rd and 4th century, mainly smaller population and replenishment problems it could and would experience if it took many casualties in battle. Thus, the Roman army or armies were even less willing to fight in pitch battle than its Punic and Augustan counterparts due to the fact that it would face more hardships in replacing losses from battle
In conclusion, the Roman army was plagued by civil war, a smaller population, corruption, lack of discipline, change of tactics, and local identity, making it difficult to fight foreign invasion.
Edit 1
I would like to further elaborate on a few points in the third paragraph. With the integration and recruitment of more foreigners into the army (especially Germanic peoples), one saw an increasing influence upon the military. First, the equipment used differed from earlier armies. The most glaring weaponry differences was the abandonment of the gladius short sword for the long-bladed spatha sword. In addition, Western Roman soldiers began to wear barbarian "trousers" instead of loose fitting tunics. Finally, some Roman infantry behavior had changed as well. As addressed above, some infantry utilized war cries in battle. Roman historians discuss soldiers employing a Germanic war-cry called the barritus which was characterized by a rising crescendo of yelling.
With these changes in tactics and equipment, some scholars argued that the Roman army had decayed into undisciplined warbands and in effect had become "barbarized." This is probably the most commonly held myth among many mainstream readers of Roman history. In actuality, the Roman army remained disciplined and the most formidable force in the Western World. The new "barbarian" recruits into the Roman army served with distinction and bravery. Adrian Goldsworthy further elaborates stating,
"Barbarian recruits were in general as loyal and efficient as any others, even when fighting against their own people. Occasionally barbarian soldiers turned traitor, but so in this period did some Romans. By the late 4th century many senior officers were of barbarian descent, yet most of these men appear to have culturally assimilated into the Roman military aristocracy^1."
Footnotes
Sources
Boak, Arthur, and Sinnigen, William. A History of Rome to A.D. 565. New York: Macmillan Publishing, 1977.
Dupuy, Ernest, and Dupuy, Trevor. The Encyclopedia of Military History from 3500 B.C. to the Present. New York: Harper & Row Publishers, 1970.
Grant, R.G. Battle: a Visual Journey through 5,000 years of Combat. New York: DK Publishing, 2005.
Goldsworthy, Adrian. The Complete Roman Army. London: Thames and Hudson, 2003.
Goldsworth, Adrian. How Rome Fell: Death of a Superpower. New Haven: Yale University Press, 2009.
Krautheimer, Richard. Rome: Profile of a City, 312-1308. New Jersey: Princeton University Press, 1980.
McNab, Chris. The Roman Army: the Greatest War Machine of the Ancient World. Oxford: Osprey Publishing, 2010.
The Roman military changed massively between 218BC and 378AD, as did the structure and purpose of the Roman state and society. In 218BC, Rome was a Republic centered around subordinating and allying defeated enemies to the city itself. The military was something that could be called up as needed, with subordinated allies in Italy being required to provide troops, and propertied Romans being recruited and divided based on property ownership. Troops had to provide their own equipment. Additionally, Italy was largely self-sufficient, not relying on regular shipments of grain from Africa or Egypt, with farms on a smaller scale and not concentrated in the hands of a few elites.
Additionally, size is not necessarily always a good thing. It can be a liability - by public transportation systems available in the period, it could take weeks or months for small groups of people to travel across the Empire. Also, having greater resources is not the same as being able to utilize them effectively. The process of Romanization had occurred by Romans giving strong incentives for elites to consider themselves a part of the Empire - citizenship was given out fairly frequently before 212AD (when Caracalla issued an edict of universal citizenship), but this meant that Roman authority continued to depend on these local elites. The chapter "The Limits of Empire" in Peter Heather's The Fall of the Roman Empire: A New History explains this pretty well.
One major factor influencing the structure of the late Roman Empire was the Crisis of the Third Century. The Roman Empire had been a semi-hereditary elective monarchy since Augustus's time, with the Senate and the military selecting and approving emperors. Starting with the civil war after the assassinations of Commodus and Pertinax in the late 2nd century, emperors had gradually marginalized the smaller, easily observed senatorial elite, in favor of the larger equestrian class which, at the time, was not assumed to be able to take power. However, the marginalization of the Senate essentially removed the main institutional barrier between the military which ensured the emperor's power, and the emperor himself. (source: Adrian Goldsworthy, How Rome Fell; Edward Gibbon, Decline and Fall of the Roman Empire, Vol. 1) According to another interpretation, to Rome's East, the weakened Parthian Empire was taken over by the Sassanian dynasty in the early 3rd century, which, with a more organized, centralized empire, was able to launch far larger, more damaging campaigns against the Romans - this forced reorganization of the Roman Empire proper to deal with the threat, if one finds the interpretation of the Sassanid Empire as a "superpower" compelling. (Source: Peter Heather, The Fall of the Roman Empire).
In the wake of the Crisis, the Roman Empire had been heavily reorganized, and it remains controversial the extent that this reorganization harmed or helped the Roman Empire. Military supply was provided by a separate civil bureaucracy, for instance. One argument holds that these reforms came about, in part, to make it more difficult for usurpers to rise up against emperors, as had been a frequent problem in the 3rd century (Goldsworthy, How Rome Fell). Additionally, taxation was expanded and restructured, so that tax collectors were perceived as oppressive (as an example: a group of Isaurians in the 4th century posed as tax collectors, engaging in coercive tactics to do so in the region, which shows roughly how such officials were viewed at the time), and so that the civil office of being a tax collector was itself increasingly viewed as a burden (exemptions from such duties were gradually lifted, and tax collectors who do not reach the required amounts have to make up for the shortfall out of their own pockets).
The military itself shifted from a static defense strategy to a defense-in-depth strategy, with limitanei keeping frontiers secure against minor and localized threats, and the comitatenses under direct imperial command existing as a mobile striking force to deal with threats. Some argue that the military itself was actually larger - estimates range as high as 600,000 troops - which would, of course, require a greater tax burden (Goldsworthy disputes this, however, asking, of 5th century military failures, where this supposedly massive army actually was). Also, given communication and travel restraints of the period, it's worth asking how effective a defense-in-depth strategy could be. One possible explanation for the shift to this strategy is that emperors needed large armies close at hand to deal with civil wars as they occurred.
Also, with the Sassanids on Rome's eastern borders, there was a constant threat which required a substantial commitment of troops at all times.
However, we should consider the fact that, despite these constraints, the Romans did fight to a significant degree as the Germanic tribes invaded their territory. However, the arrival of the Huns and economic development near Rome's frontiers largely accelerated by the presence of the Empire itself meant that there were more effective Germanic fighting forces (Peter Heather, Empires and Barbarians).
That said, 378AD wasn't the end of Rome, and had the 5th century worked out differently, would've only been a historical footnote. The arrival of the Huns drove the Tervingi and Greuthungi into the Roman Empire, where they revolted, but the threat they posed was a regional threat. It was a war the Romans couldn't lose. However, it was not easy to win, and after substantial troop losses (including the famous Battle of Adrianople), Theodosius settled the Goths as a group on Roman territory, on terms unusual to the Romans, allowing them to keep their culture and group cohesion. But it was fairly clear to the Goths that the Romans would not tolerate this forever - for instance, when a large number of Gothic Troops died in the Battle of the Frigidus River, Theodosius was praised for defeating two enemies - the usurper he defeated at the battle, and the Gothic troops in his own army. Alaric led a Gothic revolt not long after, where he made many attempts to negotiate with the Roman state for a better, more permanent deal (using violence as a bargaining tactic, including repeatedly besieging Rome) until his death. Had the Western Roman Empire not been dealing with other threats at the time, it would've likely been more able to deal with this threat.
But the Western Roman Empire had dealt with other threats. In 405-406AD, a group of Goths, as well as large groups of Vandals, Alans, and Suebi, invaded across the Rhine and Danube. While Stilicho defeated the Goths invading Italy, he was unable to immediately or quickly deal with the other threats. This led to a revolt in Britain, which rose the usurper Constantine III to power, where he moved to Gaul - only taking more troops out of Stilicho's command. Constantine III himself was unable to successfully deal with the threat, and the invading tribes managed to set themselves up in Spain. After Stilicho's downfall, the Roman state proceeded to kill off families of Germanic soldiers, who defected to Alaric and allowed him to invade Italy. By the time Constantine III's revolt collapsed, and Constantius III was able to deal with the crisis, there were already substantial settlements of Germanic tribes on Roman territory, as well as a serious loss of revenue from damage in Italy and Gaul (for the latter claim, source: Bryan Ward-Perkins, The End of Rome and the Fall of Civilization).
After Constantius III and Honorius died, a civil war raged over who would be the effective ruler of Rome - a waste of time and troops when both were needed. As a result, the Vandals in Spain were able to cross over to Africa, a place that never required the kinds of garrisons the East or the Rhine and Danube would have, and take control of it. In the time period, the Western Roman Empire's revenue largely relied on Africa, which had seen great economic growth in recent centuries while Italy and Gaul had suffered. With the permanent division of the Roman Empire (largely due to conflicts between Stilicho and Eastern Roman Emperor Arcadius's court in Constantinople, as well as a long running tendency to have two or more emperors), the Western Roman Empire could not rely on Eastern resources. This meant that sufficient troops couldn't be maintained. When the Eastern and Western Roman Empires prepared a joint assault on Africa in 440AD, a Hunnic invasion of the Eastern Roman territories forced them to withdraw their troops. By the time the Hunnic Empire had collapsed, the crisis in the Western Roman Empire had already advanced significantly and the Germanic armies within the Empire now had direct influence over central, imperial politics, and the Vandals had grown stronger - bad luck in 468AD meant that the Western Roman Empire was fated to extinction. (source: Heather, The Fall of the Roman Empire)
This account is by no means comprehensive.
Tl;dr: It's complicated - a lot of immediate as well as larger factors were involved.
There have been a number of good answers here detailing the weakening of the Empire in its later centuries, but I would like to stress a different part of the answer. Namely, the vast difference between the semi-professional militia army of the mid-republic and the professional standing army of the late empire.
In writing this answer I'm mostly drawing on Goldsworthy's "Fall of the West" and "Complete Roman Army."
The Romans of the era of the Punic War were used to raising new armies. They raised new legions every single year. (And usually disbanded them again when the conflict they had been raised for was over.) There was a system in place. Its citizens knew what was expected of them, had their own weapons and armour or could provide them for the most part. Its allies knew what was expected of them and were similarly able to raise forces on demand.
Yes, the Punic Wars rapidly drained a man-power reserve that was never meant to be tapped all at once, but the reserve was there. Polybius claims that at the time of the second Punic War, over 700,000 men were listed on the Census Rolls in Rome and its allied client states as having enough property (and the right age) to be eligible for service in the legions.
If you raised one of those republican armies, many of the men in them would have served in other legions in earlier campaigns, and therefore plenty of institutional experience was there to be drawn upon to drill the men and form them into an effective fighting force. After all, every Roman citizen of property was expected to serve a number of campaigns in his life, but otherwise return to his farm. A loss of some legions would not eradicate this experience.
Raising units in the later empire was a much more difficult affair, requiring large quantities of money that often was not available, necessitating sometimes tough searches for additional recruits in a time when many did not want to serve in the army. In the days of the Republic, service in the army marked you as a free citizen and was a source of pride despite the brutal discipline enacted by the army. But in the Empire, though there were many more citizens, this was no longer the case, and most people had no desire whatsoever to serve in the army.
This means that that large pool of partially experienced and willing man-power was no longer available. A loss like Adrianople, could wipe out an entire generation of all the experienced fighting men of certain regions. The vast majority of the people in the Empire had no experience in fighting wars, nor any culture or tradition in fighting wars, nor any desire to. (There's a notably increase in laws against draft-dodging after Adrianople, for example.)
(Though I'll stress that, as others have outlined, the "barbarisation" narrative is outdated and the shift to recruiting soldiers from the Empire's borderlands and its surrounding tribes by no means resulted in worse armies. Quite the contrary, in fact. Because these people did come from warrior cultures, they would often be much more enthusiastic and willing recruits, and just as capable as any native-born Roman once they had been trained in the Roman way of war. )
The loss of a professional army means you have to start from scratch. The loss of a force formed from a cross-section of a population that has a warrior culture means you have only lost a proportion of your strength.
The Roman Republic actually benefited from this dichotomy in their time of expansion. Many of their enemies, like the Hellenistic Successor Kingdoms, fielded similar professional armies consisting of hereditary soldier-castes. Very effective fighting forces, but as Pyrrhus and Antiochus learned, brittle. A few sharp losses were enough to beat such kingdoms.
It's not all good, of course. There were plenty of disadvantages to the militia system too. For example, in more peaceful times that experience I described would vanish, as the people who had actually fought in wars died without really getting the chance to pass on their skills to the next generation. We see that the legions perform noticeably worse after such a period of relative peace between 180 and 155BC, resulting in many battles lost by the Romans. (For example, against the Cimbri and Teutones. Though the Romans still won the wars eventually.)
Still, it is this difficulty in replacing armies of trained professionals that (partially) accounts for the very different strategic disposition of the later Imperial and Roman armies. Pitched battles become an unacceptable risk if you can't easily recover from a loss. It becomes much more sensible to use diplomacy, bribery, fortifications, raiding, harassing and other limited forms of engagement rather than stake everything on a single roll of the dice. That's the reason why later Byzantine military manuals stress these tactics over pitched battles.
"a wise commander will not engage the enemy in a pitched battle unless a truly exceptional opportunity presents itself." —Maurice's Strategikon
There are also political factors are play: Emperors legitimised their rule through victory, but in defeat they might show themselves to have lost the favour of the god(s) and encourage rivals to rise. Not to mention that losing that aforementioned army of trained professionals will make it much easier for said rivals to actually win. (See for example Heraclius) On the other hand, the temporary nature of command in the days of the Roman Republic meant that generals were encouraged to seek a swift resolution. If they didn't fight a battle right then and there, a rival might replace them and take the glory for themselves!
Still, we shouldn't exaggerate these differences either. The Romans did replace their losses after Adrianople, draft-dodging or no. As long as taxes were coming in, the Empire did always manage to keep armies in the field. They could no longer deploy the seemingly inexhaustible reserves of manpower we see in the Punic Wars, but the Empire was plenty resilient. It took centuries and centuries to fall, and even then it was a partial fall at best and perhaps better described as more of a gradual disintegration.
Actually, the Eastern Roman Empire did recover from the battle of Adrianople in 378, and went on to survive for several centuries.
The problem for the Western Roman Empire was that, throughout the 5th century, a series of almost constant usurpations and civil war, which Halsall argues were a higher priority for the Emperor than the Barbarian threat (Guy Halsall: The Barbarian Migrations) severely depleted their army. Citizenship, which had previously been a major motive for joining the army, had already been granted to all citizens of the Roman Empire in the 3rd century. Also in the Republican age every citizen was expected to take up arms in order to fight against threats to Rome, and now the bureaucracy and particularly the church provided alternative avenues for climbing the social ladder. These changes, and the loss of legionaries in civil wars (Constantine 'III' withdrew all the legions from Britannia in his own attempt to seize power, for example) meant that the Empire began to rely heavily upon Barbarian mercenary armies to fight their wars for them. When Attila was defeated in the Battle of Chalons, for example, it was a combined force of Romans, Goths, Franks and Burgundians under the leadership of Roman-Gothic general Aetius (who has grown up with the Huns) that stopped him. Some of these groups, such as the Goths were eventually granted land as a return for their service. Yet as the Western Roman Empire lost prestige and territory, particular the surprise loss of the fertile North African region to the Vandals, it could no longer afford to pay for large enough armies to defend itself or reconquer the areas it had lost. The presence of alternative barbarian groupings and warlords within the Empire provided alternatives sources of power, patronage and legitimacy, which meant that many subjects of the Roman Empire began to adopt barbarian identities (This is discussed in detail in Halsall's work)
It should be noted however, that many of these "Barbarians" would have seen themselves as Roman, and even looked to the Eastern Roman Empire for recognition and titles. Clovis I of France, for example, also sought and was given the title of Consul.