I guess the best way to explain the ancient view of pants is thus: they were not barbarian they were feminine. That is, they were both "barbarian" in the Roman sense, and "eastern."

Contrary to popular belief, pants were not adopted from the Germanic or Celtic peoples because of the cold regions. In fact, they were adopted from quite the opposite: pants came into style out of Persia, in the "hot" regions of the eastern half of the empire. The Romans adopted the tight-fitting (almost "yoga pants" tight) trousers of the Iranian world, which they called fiminalia because of the "feminine" association with the east and eastern peoples in classical thought and literature (stemming in no small part from Herodotus).

Some trousers (loose fitting, "Celtic" or "Germanic" ones) were used before the widespread adoption of fiminalia/anaxarydes in the late 2nd-early 3rd century AD. These were usually called braccae or braca or some variant thereof, which was a transcription of a Gallic word. However they were never widespread and were quickly overcome by Roman fasion which dictated the Persian style trousers be worn, and that in turn spread into the Germanic world as well. The 3rd Century AD Thorsberg-Mose trousers are in this style. Ironically, though, the actual Persian style of trousers shifted to the loose fitting Saravara or the shorter Skeleai, which weren't fitted, didn't have feet, and tended to have decorative stripes (clavi) as well.

But the average Roman would have started wearing trousers in the early 3rd century AD, in summation, although they were in use earlier.

Source:

Sumner, Graham. Roman Military Dress.