Most ancient civilizations wore robes of some kind. Horse riding and warfare forced many to wear pants and men did the majority of fighting and horseback riding. For a long time wearing pants is not universal, for example samurais would wear loosely fitted pants while everyone else in Japan still wore robes. Cultures that did more fighting and horseback riding are more likely to wear pants. According to University of Connecticut evolutionary biologist Peter Turchin, horsemen from Central Asia wore pants, the Chinese initially thought pants wearing to be uncivilized but eventually adopted by Chinese soldiers anyway.

In Europe peasant men has always wore some kind of pants for practicality. Aristocrat men wore knitted hoses for everyday wear until the 1500's when some aristocrats began to wear this kind of shorts with stockings. This eventually evolved into breeches of the 1700's. Then during the French Revolution the working class who didn't wear breeches (the sans culottes) made long pants the only acceptable men's wear.

As for why women didn't wear pants, there used to be a lot of religious prohibitions against cross dressing. In Deuteronomy 22:5 God commands that a woman is not to wear that which pertains to a man and a man is not to wear that which pertains to a woman, for all that do so are an "abomination".