It's a fraught question, and I think you've misremembered some of the details. Just to run over some of the basic details, here's how Plutarch described the problem:

Of the territory which the Romans won in war from their neighbours, a part they sold, and a part they made common land, and assigned it for occupation to the poor and indigent among the citizens, on payment of a small rent into the public treasury. And when the rich began to offer larger rents and drove out the poor, a law was enacted forbidding the holding by one person of more than five hundred acres of land. For a short time this enactment gave a check to the rapacity of the rich, and was of assistance to the poor, who remained in their places on the land which they had rented and occupied the allotment which each had held from the outset. But later on the neighbouring rich men, by means of fictitious personages, transferred these rentals to themselves, and finally held most of the land openly in their own names. Then the poor, who had been ejected from their land, no longer showed themselves eager for military service, and neglected the bringing up of children, so that soon all Italy was conscious of a dearth of freemen, and was filled with gangs of foreign slaves, by whose aid the rich cultivated their estates, from which they had driven away the free citizens.

Note that is the traditional account; it is firmly embedded in the ancient literary sources, but it's being questioned by modern scholars for reasons I'll get to in a moment. So note Plutarch doesn't say the problem is a lack of population; there are plenty of people, but they've been chased off their land by the rich and are refusing military service. According to Plutarch, Ti. Gracchus in 133 BCE (so, some 70 years after the end of P2) was the first to address this problem, and his solution was land redistribution: seize the land illegally held by the rich and redistribute it to the poor in such allotments that they would qualify for service (military service had a land requirement still). That implies there's enough landless poor to populate the redistributed land. The rich didn't put up with this and ultimately beat Ti. to death in the forum.

Ok, what does this have to do with P2? Brunt in his book on Italian Manpower drew up a list of reported census figures between the first census taken supposedly in 508 BCE and the last Augustus took in 14 CE; that's 40 reports. For the period around P2 Brunt gives these numbers. In this chart the first number is the year and the second is the number of citizens counted.

234/3 -- 270,713

209/8 -- 137,108 (Brunt suggests this could be emended to 237,108

204/3 -- 214,000

There is evidence that Rome was hard pressed for soldiers. Ti. Gracchus, the grandfather of the Gracchus I already mentioned, enrolled an army of slaves, a very un-Roman thing to do. The only other instance that comes to mind is Marius when he returned to Rome after Sulla's coup.

If we use the numbers as reported, the first half of P2 cost Rome about half its citizen population. However (and Brunt doesn't say so where he put this chart. He might later, but I bet this is the reason for his emendation) the number for the 204 census shows quite a sudden recovery. The emended number makes a steady slope in decline. In any case, between 234 and 204 some 50K citizens have dropped off the rolls.

Then there are questions about what the census numbers mean. How accurate are the numbers reported? How accurate was the census conducted? Who were they counting? All citizens? This would include women and children. Men eligible for military service? Heads of households? Did Romans always count the same thing, or did they count different people at different times? Different scholars have argued at some point or another for each of these, and it's not a settled debate yet.

Anyway, back to Hannibal. Toynbee in 1965 thought Hannibal absolutely devastated Italian agriculture. Even if this were right, Brunt says this wouldn't have affected Roman agriculture. I'm not sure about that point; Gracchus' land reforms were a pain in the ass for the Italians, because Gracchus' surveyors of Roman lands were accidentally redistributing Italian lands to Romans. If Roman and Italian lands were so intermixed that the surveyors could screw it up, I doubt Hannibal would have cared very much. In any case, we know from Polybius and Livy that Rome was importing grain from places as far away as Egypt, so while the crops might be lost, like the Athenians in the Peloponnesian War, the population could be supported inside the cities.

I think I side with Brunt on this. By 174 the census numbers are back up in the 270k area. Whatever damage Hannibal did to the Roman population was repaired in a generation. So what caused the collection of land by the rich, the new masses of urban poor, and the lack of military manpower? After P2 Rome almost immediately embarked on a series of wars in Spain and in Greece that didn't last a season or a year, but decades. The traditional view has Roman yeomen (for lack of a better word) who qualified for military service spend years away from their farms and families. While these men were overseas their farms fell into disrepair; the families sold the farm to the rich who populated them with a new labor source of slaves supplied by the overseas wars; with nothing else to do the families moved to Rome where they could live off the grain dole and at least the men could vote.

Rosenstein has recently challenged this position with comparative evidence from other preindustrial societies. Turns out women are perfectly capable of running farms on their own, and the archaeology (I don't know how you tell this. /u/tiako help?) doesn't support consolidation of plots. Rosenstein suggests that it wasn't a lack of bodies that cause the military shortage, it was war exhaustion. Men did not want to spend decades fighting overseas after having spent two decades fighting in Italy. There is some evidence for this. Livy reports that when the Senate declared war on Macedonia in 200, the Senate could not get a declaration of war from the assembly :

The proposed declaration of war against Macedonia was almost unanimously rejected at the first meeting of the Assembly. The length and exhausting demands of the late war had made men weary of fighting and they shrank from incurring further toils and dangers.

The Senate got its declaration with this compromise:

The consuls were ordered to disband the old armies and, each of them, to raise two fresh legions. As the conduct of the new war, which was felt to be a very serious one, was entrusted to Sulpicius, he was allowed to reenlist as volunteers as many as he could out of the army which P. Scipio had brought back from Africa, but on no account to compel any of the veterans to join against his will.

So if there were still small farms and still eligible if exhausted soldiers, where did the masses of urban poor come from? That goes back to the census numbers and what you make of them. There is an oddity with the Augustan census figures. The census of 70 BCE counted about a million citizens. The next one, in 28 BCE, four million. Brunt thinks Augustus must have changed what the census was counting. Originally the census was to enlist male citizens in their proper voting blocks and to qualify people for the military. But since Marius enlisted the urban poor, the wealth requirement no longer mattered. Brunt explains the giant jump as Augustus deciding to count all citizens, not just those males of military age. He estimates the total free population of Italy under Augustus at somewhere around 6 million.

Lo Cascio in 1994 (so, almost 25 years after Brunt) points out there's no literary evidence that Augustus changed the practice of the census, and that Augustus was still just counting males of military age. That would put the total free population at somewhere around 12 million, which he thinks is beyond the carrying capacity for Italian agriculture in a preindustrial society. The masses of urban poor existed not because they had been driven off their land by the wealthy, but because there was no more land available for the expanding population.

TL:DR P2 had a significant effect on the direction of Roman history, but not (in my and Brunt's opinion) because Hannibal caused such devastation on the countryside and the population, but because Rome was not prepared to support the decades of war that directly followed P2.

And this is what I do when I can't sleep on a Friday night/Saturday morning. Just shoot me.