This is a very complex question to answer. There are a lot of possibilities but we don't know precisely why. Many solid and well supported theories are covered in this thread: https://www.reddit.com/r/AskHistorians/comments/470bpp/during_the_punic_wars_rome_and_her_allies_lost/

Edit: at the request of the mods, /u/Goodmorningdave, /u/nihilistsocialist, /u/iguana_on_a_stick all have comprehensive answers covering a variety of factors such as, the crisis of the 3rd century, plagues, economic crises, cultural shifts, and the increased activity required of the military in the late empire.

I wonder what effect the Marian Reforms had on this subject matter. But I'm guessing the outstretched empire needed more logistical manpower, than fighting men. Logistics never gets enough credit. Basically, there were not enough fighting age men available and willing to be conscripted.

Two piggyback questions:

1. It seems that the armies fielded during the Social/Punic/End of the Republic wars were of much larger size than nearly all armies until the late middle ages. Why is this? Was it that there just weren't players as large as Rome fighting?
2. In the 500s and the 900s, Western Europe was swarmed by tribes, raiders, invaders, and migrations of people from the North East -- I would think that the warmer West and South would be home to many more people (longer harvest periods, easier living), but the sense I get is that the numbers were on the side of the Germanic tribes/Scandinavian raiders? Is this wrong/am I conflating numbers with 'hardness' of the soldiers?

One major factor is being a soldier just became a shitty job.

In earlier eras, Roman citizenship was quite limited. You had to be a native born Roman, in Rome itself, and no one else in the empire had the same privileges as the citizenry. This limited your options. The military became an attractive job option for many people because of the benefits. I'm going to be general because this is a long period of time and things change, but let's say you decide to join up in like, 50 AD. You serve for 25 years and get the following:

A) Steady pay. Not to be underestimated. B) A secure food supply. C) The best medical care in the ancient world. D) Possibility of advancement to positions of authority and responsibility. E) Upon discharge, you get citizenship (as do your children that you're not supposed to have but, let's be realistic), and are supposed to get a lifetime pension and land. This doesn't always happen.

If you're some random dirt farmer, it's not a bad deal.

As time progresses, the benefits dry up. Citizenship is first expanded to everyone in Italy, then in the 200s AD it is to the entire empire. The pensions and land grants dry up because there's not much land (usually, veterans were settled on the frontiers but once the empire stops expanding, there aren't new frontiers to settle), so as jobs go being a soldier starts to suck. Fewer and fewer people join the military. The Romans don't appear to have had any sort of conscription until late antiquity, when many jobs become hereditary--including being in the legions. So you have these people who are forced to join the military because their fathers were soldiers, but it's not a huge force. Recruiting beyond that is nearly impossible. However, there are all these Germans who really want to be part of the empire and are happy to fight in exchange for benefits, so a ton of recruiting is done among the Germanic tribes.

One other factor in the Punic Wars, the second particularly, is just that Rome didn't follow the rules. In the ancient world you were supposed to field armies, have a battle, and the losing side sues for peace. The Romans never, ever stop fighting or agree to unfavorable peace terms after the Gallic sack of Rome in 410 BC. Hannibal can trash Roman armies over and over, but they just won't give up and make peace like they're supposed to.