Tacitus, in his Germania, specifically referred to the Germanic people as having huge frames:

All have fierce blue eyes, red hair, huge frames, fit only for a sudden exertion.

Allthough it is important to remember that Tacitus never was in Germania, but rather based his reports on reports given to him by soldiers.

Another piece of evidence that suggests a height difference between Germanic people and others around them is the difference in diet. The Germanic people had a significant portion of meat in their diet, which is condusive to (via attaining easier calories and protein) growing and muscle building (a modern example of this would be the introduction of a western meat, mainly beef, based diet to countries like Japan; The average height shot up).

Based on actual skeletal remains, we can see that Northern Europeans (Germanic People) and Western Europeans (Gauls/Celts, I am including Britons in this) were pretty much the same height, with the Germanic people maybe a centimeter or two taller, and the roman people were on average shorter.

But it is important to remember that these are averages. The average Roman soldier would still tower over the average Germanic man.

As u/Boccard mentions, the size of the Germanic and Gaulish peoples is a well-known trope in Latin literature, and research on skeletal remains of the period backs this up.

However, there are a few more factors to consider. Firstly, by the time of the Empire, legionary recruits no longer tended to come from Italy. The eastern legions were recruited locally right from the start, and the proportion of Italian recruits dropped very sharply very quickly even in the West. Around the mid 1st century B.C., it's been estimated that only 22% of legionaries were Italian. In the second century, this dropped to a mere 2%.

So when you're imagining the Roman legions on the Rhine or Danube, you should imagine legions recruited largely from populations of in that region. Many would be descendants of Roman soldiers who had settled in the frontier regions and married local women. Others were recruited from local people such as Gauls who'd gained citizenship. In the east, it was common practice to recruit local people and grant them citizenship on enrolment, to satisfy the "citizens only" requirement. In addition, many auxiliary units consisted of Germanic peoples themselves.

All this may well mean that there was a bigger difference between, say, a Roman legion on the Rhine and a Roman legion in Egypt than there was between that Rhine legion and a German tribe across the river.

Secondly, however, an interesting conclusion coming from recent research is that average height dropped during the time of the Roman Empire, compared to what it had been before or had been after. Mean male height in what is now Italy indeed became as low as 164cm, just as low as it would be in the 19th century, which is when it was the lowest it has been in recent history.

This probably had to do with a combination of population increase (Malthusian pressure) and taxation reducing surplus and reducing variety in food for the poorer classes. There's not enough data available to tell how this differed in richer or better fed classes. (But soldiers were recruited from the poor, and whilst their rations would have been plentiful, height is determined by the food you get in childhood and adolescence.)

Germanic peoples in what is now Germany would most likely have had far lower population densities and a more protein-rich diet, so it's likely there still was a considerable difference in average across the border.

• On recruiment and the army, see Scheidel 2007, Roman population size, himself drawing on Italian research)

• On average height in the empire, see Scheidel 2010, Physical Wellbeing in the Roman World