First of all: It is very hard to guess the number of combatants. In a typical medieval battle, the contemporary sources will have grossly inflated numbers that are just too incredible to be taken seriously. For example, William of Poitiers claims that the Normans brought 50,000 men to Hastings. That’s ridiculous. The chronicler probably wanted to make the battle more memorable, but it is annoying nonetheless and historians have to come up with “guesstimates”. I do not know whether your Roman invaders and the British received a similar treatment, but quite frankly, when I hear “150,000 combatants” it raises some flags which also reflects unfavorably on the “45,000”. The Roman writers may have exaggerated the number of enemies they faced in order to boast. Or these were levies/civilians/tribesmen who were conscripted to make the army look bigger, instead of professional warriors, but therefore offered limited martial value.

For the battle of Hastings, you are on the “low-counter” end. William’s army was small, but not as small as 5000, a commonly called up number being 7000 men (Morris: The Norman Conquest). The Saxons, despite fighting in their home and potentially drawing from an entire kingdom, probably merely reached similar numbers. Earlier, Harold had already disbanded his army because he thought the campaigning season was over, only to be promptly attacked by the last ‘Viking’ invasion in English history, which diverted the English’s attention. According to Morris, the time Harold took to go back from northern England (Vikings) to southern England (Normans) indicates that Harold’s army travelled by horse, meaning that he left the foot soldiers of his army in the north. In London, he rallied what troops he could find in southern England (in addition to his mounted troops) and then marched to oppose William. If he had not had disbanded his army earlier, or been distracted by another invasion force, Harold most likely would have had substantially more men at his command.

Another important issue with your comparison is, that you are comparing a vast Mediterranean empire to a singly duchy in northern France. Imagine if the USA broke apart into independent Waring States. Of course the army that Rhode Island can field will be a lot smaller than that of the former USA proper. It would be a less biased comparison if you compared the Roman Empire to, say, the crusades, as these could draw on forces from an area of a more similar size.