There's always more to be said, but I have a previous answer that addresses how men wore "suits" as everyday dress in the past, which I'll copy/paste below:
It's not that casual clothing wasn't accepted, it's that they just considered different things casual. What you have to understand is that while there was something of a "casual revolution" from the 1930s through 1960s, both the later twentieth century proliferation of denim and the very late twentieth century proliferation of knit fabrics would go on to create a very different image of what "casual" inherently looks like. That is, we think of blue jeans, thin t-shirts, sweats and sweaters, leggings, etc. Informal clothing existed, separated from formal clothing sometimes by cut (formal evening or "full" dress was typically made from different patterns, with tailcoats for men, short sleeves for women, etc.) and more typically by being made out of less formal materials.
So the men in photographs wearing what looks to you like "business clothes" are generally in casual clothing that they wouldn't think of as formal suits. Let's take a gander at a few photos.
Subway Construction Workers, 1901-1905 and Lenox Ave. and 140-141 Sts., 1902
The men here are wearing collared button-down shirts and trousers held up with suspenders (braces if you're British). The subway diggers have broad-brimmed hats and the street workers are probably wearing something more like a fedora (the view is obscured). In a modern context, that would be non-casual - but in their own context, this is casual dress for manual laborers. Most of the shirts are dark or at least colored, a practical choice for dirty work in comparison to bright white; some of the men in the subway also seem to be wearing coats, which would add an extra layer of protection. The coats and trousers were most likely made of a sturdy cotton or wool twill, which would be durable. The hats could help keep dirt and debris away from their faces in all cases, and some of the ones in the subway photo appear to be made out of a hard, shiny material.
Please note that I'm not saying that protection was the main reason for wearing all of these things, with the implication that people stopped wearing them when they no longer needed protection - my point is that they didn't get in the way, that the standards of "appropriate" adult dress were adaptable.
Exterior marble work : two workers, 1905 and Exterior marble work : men flanking a piece of marble being hoisted, 1905
More coats, trousers, and hats. The hats are especially useful to these men, being outside and in full sun. You can see that the jacket of the man on top of the building is worn and dirty because it's a working garment; one of the men on the ground (who is wearing overalls, not trousers) took his off and rolled up his sleeves, evidently not needing to be as protected from physical harm as much as he needed to be cooler - he might be more in charge of the mechanism for elevating supplies than actually chipping away at marble himself. The only man actually in anything relatively formal is the one on the far right in the second photo, and he's quite likely the foreman, more of a manager than a laborer.
Modern dress pants are typically made to look good, not to wear well while stooping, kneeling, etc. because they aren't being worn to do manual labor. But there is no real reason that pants have to look exactly like our modern blue jeans to be good for working in! What makes jeans durable is just that they're cut from a twill-woven cotton fabric; the blue dyeing process is irrelevant. I want to stress that these were not necessarily "dress pants", just because they were being worn with suspenders and a jacket of some sort: modern men only add these accessories if they're dressing up (well, not all of them - plenty of old farmer guys wear suspenders on a regular basis) because they now denote formality, but that is a relatively recent development.
So there wasn't the same concern over "ruining a suit" of work clothes that you would feel about "ruining" a dress suit today. What you think of as a "suit", they would have thought of as their "Sunday suit". You - the Edwardian version of you - want to take care of your work clothes, have your wife brush them, patch them, reattach the buttons, etc., but it is not as important as your Sunday best, which is only worn to church and on special occasions (the equivalent of your modern dress suit). Does this make sense?