Average men in pictures from the 19th and early 20th centuries are usually seen wearing what would be considered today as formal dress (suits and blazers) even in their everyday lives. This seems most striking in photographs of men standing in line at soup kitchens during the depression wearing full 3 piece suits and hats.
In the 21st century, even lower end suits can be significantly more expensive than the clothes that most average people wear on a daily basis. And nicer suits can cost as much as an average person's monthly rent. Or more, if they're custom fit and properly tailored. The result being that most men who are not required to wear a suit to work will only own one or two reserved for special occasions.
In the past, were suits comparatively more affordable? So that an average person could afford to have a variety of suits and not appear to constantly wear the same outfit. Or were average people simply expected to invest a significantly larger portion of their income into a smaller but more formal collection of clothing to conform to traditional fashion?
This previous thread very close to this topic has an answer from /u/chocolatepotmay that may help.