The question is answerable, but only if you avoid conflating historical reality and myth. In your question you mention a historical population of around 10,000, and a legendary figure of 100,000 Greek troops.

At least, that's what I infer: the 100,000 figure would normally come from looking at the Catalogue of Ships in Iliad 2, looking at the total of 1186 ships, and the figures of 120 and 50 crew on each ship for the Boeotians and the southern Magnesians respectively, averaging them to 85, and getting 100,810.

The legendary war is legend and doesn't need to stick to realistic figures. If Homer's description implies the Greeks had 100,810 troops, then 100,810 is the number of troops that they had. Homer doesn't give figures for the population of Troy or the army on the Trojan side (though bear in mind that he gives them allies from all over western Anatolia, from as far afield as Lycia).

As to history: this side of the question is more speculative, since there's no evidence of a war between Troy and any mainland Greek states at any date, ever. It's purely a thought experiment -- 'If something like a Trojan War ever happened, how many people might have been involved, given that we imagine the war happening at such-and-such a date?'

At the time the Iliad was composed, in the early Archaic period, the armies would certainly have been much smaller, perhaps on the scale of a thousand if you're lucky. Population wasn't high: even though there is reason to think there was a population boom going on at the time, population certainly wasn't at 5th-century levels. Looking back to earlier centuries, it would have been even lower: at any time in the previous 500 years, you'd be looking at something more like a viking raid than a large war.

To get a large army you'd have to imagine a war before the end of the Bronze Age, when the Hittite and Mycenaean palace cultures were still functional. (As it happens, Classical- and Hellenistic-era Greeks like Herodotus and Eratosthenes did come up with guesses for the date of the legendary war that fall in the Late Bronze Age by modern reckoning, but their dates are purely guesstimation: they had no notion that there had been a substantial Mycenaean palace culture, or a Hittite empire.)

We don't have evidence on the size of Mycenaean armies. We do have some indications from the Hittites and Egyptians at the time, though. The clearest evidence comes from the battle of Kadesh in 1274 BCE, when Hittite forces under Muwattali II engaged Egyptian forces under Ramesses II (and entertainingly, both sides claimed victory). An Egyptian report suggests the Hittite forces numbered around 47,500, including a huge number of mercenaries. This could be an exaggeration to explain why Egyptian territory suddenly receded an awful lot after the battle, of course. Then again, the hittitologists Richard Beal and Trevor Bryce think that the figure is probably in the right ballpark: Bryce ends up estimating 40,000 Hittite soldiers and 10,000 mercenaries (Warriors of Anatolia, p. 168). Kadesh was an exceptionally large engagement, so this is certainly not typical: Bryce suggests about 10,000 for a more 'normal' Hittite expeditionary force, and around 55,000 for the total combined forces of the Hittite empire at its height.

That's still a long way from the legendary figures that Homer implies. But then again, it is legend. It's pretty clear that, Greek population levels in the 7th century being what they were, the whole point of the Catalogue of Ships is to impress you by how impossibly vast the Greek forces are by comparison with anything that was remotely possible at the time. It doesn't have to be real, and it's pretty clearly not supposed to be real.