Well the concept of fashion clearly existed even at the time of Julius Caesar - and more specific to your question, so did the generational divide with regard to fashion.

Julius Caesar is said to have worn a loose belt over his senatorial tunic, which in his case had long fringed sleeves. These fashions seem to have caused comment among his more conservative Roman contemporaries. Sulla is said to have remarked of him: 'beware the badly belted boy' (Suetonius). Cicero made jokes about Caesar's long sleeves, and this wasn't an uncommon line of criticism. Cicero had attacked the supporters of Catiline as the sort of men who dressed in long sleeves snd ankle length tunics (Cic, Cat. 2.22) In 142 b.c. the censor Scipio Aemilianus attacked P. Sulpicius Gaius for his long sleeves - taken as a sign of effeminacy.

If anything, the toga had become unpopular long before the end of the Roman Empire. All in all it was a heavy, hot, ridiculously difficult article of clothing to put on and wear and Roman men were reported to hate it (think of many men's attitudes to having to wear suits at formal occasions today). The emperor Augustus, as part of his drive to restore Rome to a prior glory, enacted laws that forced the wearing of the toga in the Forum and surrounding environs, as many men had taken to trying to get away without wearing it.

So in short, Rome definitely did have the concept of fashion and men (at least of senatorial rank) were judged by it. The toga would have likely fallen out of use naturally during the early empire if its use hadn't been enforced for political reasons.