A study by Ray Laurence on skeletal remains in Pompeii and Herculaneum (so of city-dwellers, who may have had a different diet and therefore slightly different height than the farmers you specifically ask about) found an average height for males of 166 (Pompeii) -169 (Herculaneum) cm. A study by Kron on 927 adult male skeletons from 500 BC to 500 AD (not sure on the distribution of the skeletons through these years) found the very similar number of 168 cm (5'6") for adult men, on average. Laurence notes that this is actually slightly higher than the average male height for the city of Naples (same location as Pompeii and Herculaneum) in the 1960's.

I don't know for sure where to find data on the English peasant -- not my area, I apologize. I recall reading about a study on nutritional differences making some small difference between peasant and noble heights in Medieval England, but I can't help much more. A quick search did give me this article, which may or may not be any good, isn't confined to England, and doesn't give specific numbers in that digest on 1400 AD. It does mention a decline from 173 cm in the early Middle Ages (ca. 9-10 century AD) to 167 cm by the 17th-18th centuries), so somewhere between there, and that includes Scandinavia.

By way of comparison, the average male height in the US is 177 cm (5'10") now.