I have never been able to fully wrap my head around this aspect of the decline of the Roman Empire. The Republican era saw the Romans, with a much smaller population, suffer catastrophic defeat after catastrophic defeat (Lake Trasimene, Cannae, Arausio). Each time they replaced the losses and they won out it in the end. In fact, abundant manpower from allies is often cited as a driver in Roman's ascent to Mediterranean hegemony.
Yet the army seems to have become very brittle in the later years. A defeat like Adrianople, which was small in comparison to Cannae et.al. is credited with crippling the Roman Army for decades afterwards.
So what gives? If the answer is a change from a citizen levy to a professional army, then it seems that Rome made a mistake in moving from a successful system that could mobilize endless numbers to one that required years and years and years to get back up to strength after a defeat. Was the later army that much more elite and effective than the Roman Republican legions that won Zama, et al? And, as far as I can see, there wasn't really a truly catastrophic and irreversible decline in tax revenue until the Vandals invaded North Africa, which would seem to work against any hypothesis based in a lack of funds.
Yes, what you're seeing is mostly the difference between a militia army drawn from a wide cross-section of the population, and a professional full-time force, serving for decades at a time, paid and equipped by the state.
Nor are you entirely wrong in your observation that the army of the Republic was excellent at waging wars of conquest and winning pitched battles and that from this perspective there would have been no reason to change things.
However, there are a few things to bear in mind.
States and institutions evolve over time. Even if nobody really wants to shake things up dramatically, changes happen. The Roman state lasted for centuries upon centuries (depending on how you measure it, up to 2000 years) and during that time it went through extremely drastic changes. It is to be expected that the army would also change.
Rome's militia system worked because it was the system of a small city state, where all the citizens were directly involved in the politics of the day, and where military service was considered to be an essential part of what it meant to be a respected citizen. The people who fought in the army (or at least their fathers) were also the ones who could vote in the assemblies. They came from a culture where it was expected that a free citizen would own the weapons and armour required to fight in this army, and presumably had at least some practice using them. They would go to war besides their neighbours and relatives, and come home to live among others who had had similar experiences in the past.
The non-Roman parts of the army for the most part came from very similiar communities allied to Rome, and would fight for similiar reasons for their own communities.
It was not possible to mobilise such large percentages of the population in a multi-ethnic, far-flung empire where no such idea as the nation-state existed. The social structures and conventions that motivated the Republican Romans to march to war year in year out simply did not exist in the much larger, much more impersonal later Roman state.
Nor was it logistically possible to coordinate such a thing empire-wide with the resources of the Roman state. In Rome, the people could be mustered because they all lived in the same region, shared the same culture, and enforced obedience through social bonds and conventions. Spanish and Syrian subjects of the empire had no such motivation. Nor did the later Romans themelves, because...
The army of the principate served a very different role from the army of the republic. Even if the manipular legion was excellent at winning pitched battles, it could not have served in the same way the later, professional Roman soldier did.
The soldiers of the Republic would expect to fight for a few years only. They wanted to win glory and acclaim, and hopefully a nice bit of loot for themselves. They did not want or expect to spend their lives under arms. It was easy to get an army to defend Rome against Hannibal or to humble the king of Macedon and loot his treasury. But it was a much harder sell to help Augustus subdue the Spanish hill tribes, let alone to go garrison a wall near Scotland for 20 years.
There were always exceptions, of course. Long before the army turned fully professional we see evidence of a growing class of professionalising soldiers, men who did spend most of their lives fighting in Rome's armies. Presumably these were the men who had a taste for it, who were succesful at it, and who managed to improve their lot in life through it. Through men like these the army was already starting to change while Rome itself and the wars it fought were changing. (The exact process is a matter of considerable debate and a different question)
In the Empire this had become the rule, rather than the exception. The army was an excellent source of upwards mobility for those sufficiently well connected to make a career of it (patronage remained an essential part of the Roman way of life) and a decent enough living for those who remained in the ranks. Rather than the people of a city state under arms, it became a culture of its own, drawing its recruits from veteran family and the peoples on the frontiers. Already after the first century, the role of Italians in the army shrank to almost nothing. Rather than a means to an end, the army became an end of itself for the people who served in it.
Motivation aside, professional army had a very different job description than the militia-army did.
It is questionable how great a NEED there was in the early empire to have such a large force stationed at the borders full-time. To be sure, there were plenty of threats to Rome, and plenty of wars to fight, but as Guy Halsall points out: for almost all of history before or since states and peoples seem to have gotten by just fine without keeping hundreds of thousands of soldiers under arms at their borders. Did Rome really need 28 legions to stop German cattle raids and the occasional bout of Parthian adventurism?
The primary function of the legion was to ensure the emperor's power could not be challenged. The primary function of the Roman state was to pay for the legions. (If this seems a bit circular... well. Yes. Particularly once we reach the stage where emperors themselves are produced by the legions.)
But the legions did much more. The early empire in particular was practically ungoverned. There was no bureaucracy to speak of. The emperor operated with a tiny staff of household slaves and secretaries. Provincial governors relied mostly on their own resources when going out to rule the provinces. But the army provided a giant pool of trained, reliable manpower. The army worked at great construction projects. It escorted and assisted officials. It kept track of what was going on inside and across Rome's borders. It helped with tax collection and law enforcement. It spread Roman culture and Roman identity.
None of this could have been done by a part-time militia force.
In the later empire, the Roman state did develop a larger civilian bureaucracy and the army took on a more purely military role. (at least where the field armies were concerned.) By then all memories of the Republican system were long-gone, though, and besides the other reasons why a milia system would not work remained in force.
Nor should the professionalism of the army be discounted as an advantage in war. Yes, the manipular legion had at times been amazingly effective. But it did not have the later legions' experience as siege engineers and fortification experts. Nor did it have the institutional continuity that kept the army effective over generations. Republican armies generally needed time to train together and get used to one another before they became effective, and because they depended on ad-hoc recruitment it was always a matter of luck (and how many wars there were) whether veteran soldiers could be had to teach the newcomers. The professional army was much better at passing along its institutional experience and integrating new recruits.