I would say that it is indeed a reaction to the former. During the 19th century there was a big reactionary shift from the bright colours, frills, and clean-shaven faces of yesteryear, as the colourful fashions of the 18th century were increasingly associated with excess and effeminacy, in particular with effeminate dandies and macaronis (although one of the greatest Regency dandies, Beau Brummell, was actually the one who made trousers popular). Hence the shift to more drab colours in male fashion, as well as the increased popularity of facial hair such as sideburns and thick, full beards.