Hi, I am not a historian, but rather an epidemiologist who studies human growth. So, I cannot answer your question, but hopefully I can offer a few pieces of side information that might be relevant.

First, was there a height difference: answer is probably yes, since adult stature is associated with socioeconomic status very commonly, in societies ranging from Bolivian hunter-gathers to Finns.

Second part of your question: were these differences noticeable? (Even today, native-borne poor people of your own country are probably a little shorter than wealthier ones, on average - do you notice it?) Statistically, difference in height between richest and poorest, in modern populations, can range from as little as ~0.5cm to as much as ~5cm or more depending on the source (plus the two sources above, + JM Tanner Fetus into Man). But to compare, it's also useful to think about the standard deviation in adult stature, which is around ~6cm-7cm (same citations again). So you might imagine two bell curves, heavily overlapping, but one peaking to the right of the other- in other words, in any group it's very likely that you will still observe short rich people and tall poor people regularly. Unless you go out and start measuring, you might never notice that the rich people are, on average, just that bit taller than the poorer ones... (or you might notice! I'm not sure, my point is just that it's not, necessarily, intuitive.)

Third part of your question (implied) -were these differences due to different diets or better foods? Yes but not entirely. Besides diet, other factors that relate to adult height that are pretty important and likely apply include parental height, and exposure to infectious pathogens. Parental height is important because it takes generations for a population to get tall - for example, it took from 1883-1968 for the Swedish population to increase an average height of about 8 cm (Tanner again). And infectious diseases are relevant because, in societies where hygiene is sub-optimal, children face a high burden of growth-suppressing infections and parasites. So within a society wealthier parents might be slightly taller to begin with, and their kids end up a little less exposed to illness, and they might (or might not, because, you know, sometimes rich people do dumb things like not breastfeed their kids as much) have slightly better diets - kids end up a little taller. But to extend the comparison a little further, imagine comparing the population of, say, Denmark, to the population of Bangladesh. You see a big height difference, but that represents the cumulative effect of multiple generations of completely different diet, improved water, and improved sanitation - so those differences are probably going to be much bigger than what you would expect to see at two extremes of a single society, where one extreme is able to mitigate a bit better than the other but both are still facing roughly similar conditions.

That said, once people did start getting into the measurement of human height -I'm not sure when this started, but one famous early study of human growth was done in 1758, by Count de Montbeillard- it then appears to me that anthropometry gets picked up, rather soon and rather unfortunately, by eugenics-type thinkers - at this point I am completely out of my depth, though, so I will end on another question: were comparison of height between populations or social groups used by eugenicists to inform or justify their ideas?