That's one of those ancient questions we don't know the exact answer to.

However, this article came across the front page of /r/Archaeology a few days ago.

First pants worn by horse riders 3,000 years ago: Oldest known trousers originated in Central Asia

The title is a little sensationalist and click baity (first and oldest known are not always the same) but the article is pretty interesting.