I was wondering why the Empire had such a difficulty raising troops even though it had at its disposal a much larger pool of available men. For instance, I read Adrian Goldsworthy's Augustus a few months ago, and I remember him describing a revolt in Illyria, in which Augustus had a hard time filling his legions to full strength due to lack of manpower, and if I recall correctly, he resorted to freeing slaves to fill the ranks.
Another interesting thing is that during the civil war, the total legions of the triumvirs numbered over 60. And this seems to have been the amount of legions in service during the days of Caesar as well. Yet once Augustus was the undisputed ruler of Rome, he cut the amount of legions down to 28, and for the rest of the Empire, this was usually the amount of active legions.
A perfect example of this happening was during the 3rd century B.C. when Rome suffered some crushing defeats, like when Hannibal slaughtered 70,000 Romans in a single battle, and yet they simply raised a new army like nothing had happened at all, even though they only recruited from rich citizens who could afford to buy their equipment. I simply don't understand how the Empire only had about half the amount of active legions and more available men, yet had more problems with manpower.
In general periods of peacetime, the Roman Empire, in both the Principate and Late Antiquity, did not have fundamental difficulties in raising troops, nor suffer significant manpower shortages. It is erroneous to compare the size of armies of the Republic with those of the high/late Empire because these are fundamentally two distinct military systems, as the army of the early-mid Republic was a militia force, while that of the Empire was a professional one.
To simplify (as I am not an expert on Republican arrangements), the army of the Republic was a militia. All male Roman citizens (as well as the citizens of allied Italian cities) could be drawn into service, with distinctions based upon age and wealth, as individuals supplied their own equipment (thus poorer, younger men acted as velites, or skirmishes, while older, wealthier men formed rear formations). As a militia, soldiers were non-professional; they served for the duration of a campaign and returned home. Thus, the Roman state could swiftly raise large forces (as you mentioned in regards to Cannae), as a broad pool of men (both citizens and allies) were legally liable for service.
Professionalization of the army is largely completed by the Late Republic (why the army evolves from a militia into a professionalized force is complicated and not directly relevant), but it's Augustus who institutionalizes it under the control of the state (and, most importantly, the emperor). Augustus and his immediate successors transform the army into a permanent force, stationed along the frontiers, staffed by long-served professionals (terms of service ranging from 20 to 25 years). The armies of the Republic had been ad-hoc, temporary forces of citizens-soldiers. Now, the army was a permanent force of long-service professionals, clothed, equipped, fed, and paid by the state.
The cost of the army was it's true limiting factor. The professional army of the Principate and Late Antiquity was monstrously expensive, consuming roughly half of annual state revenue. Although the size of the army increases during the fourth century (this is a hotly debated topic, a middle-of-the-road estimation is a paper strength of 350,000), the extreme cost of the army placed an effective ceiling on the number of soldiers that could be afforded. Political considerations also limited size, as emperors in all periods monitored closely the number of troops commanded by subordinates.
It's important to remember that the strength of the professional army never really lay in numerical advantage. Due to it's professionalism, the army held a distinct advantage in training, discipline, unit cohesion, and equipment over their foes, as well as it's ability to remain supplied in the field in all seasons. None of the Empire's opponents, in both the Principate and Late Antiquity, could match this organizational sophistication.
Issues regarding military pay, the desirability of service, and (especially in Late Antiquity) the role of conscription remain hotly debated topics. Yet in neither the Principate nor Late Antiquity did the army face a manpower shortage due to demography. The army of the Empire was smaller than that of the Republic because it was (organizationally speaking) an entirely different force, existing in an entirely different context. This is absolutely critical to remember whenever discussing either force.