In the 1800s, Napoleon was actually about average height, and people were shorter on average. Were people from ancient times significantly shorter, or uglier because of selection of taller, more handsome mates over time leading to taller or more handsome?
I'm pretty sure differences in size and stature have more to do with proper nutrition than natural selection of some sort