What you see in photos pretty much sums it up. You'll find all manner of people being photographed, especially after the CDV process and dry plate technology of the 1880s made photographs affordable and photography a very accessible hobby respectively. You can find tradesmen posing in work clothes with proprietary tools, people in lower classes being photographed and people of higher station all sporting their chosen garb. One particular photograph of my paternal grandmother's side from the last quarter of the 19th c is a family portrait taken standing in front of a hung bed sheet on the side of their church in rural Alabama. It's evident they wore their Sunday best, which didn't include shoes however.

The difference between suits, big dresses, and casual clothing from your question probably comes down to fabric choices. Laborers or folks of lesser means still wore suits of clothes and big dresses. Instead of a fine wool broadcloth frock coat a working man might be wearing a wool jeans cloth sack coat or a paletot whereas a woman is probably wearing a cotton dress over a corded petticoat rather than taffeta over a crinoline. A banker is most likely wearing a fine tailored frock in winter, but also wearing a unlined linen suit in summer just like everyone else. Now not every man might be wearing a silk waistcoat underneath that linen summer coat, as others may be wearing a cotton one, just like some women will have the silk dresses heavily trimmed and laced bonnets while others simply have a woven one pair with a cotton gown.

Either way most every man is going to be wearing a suit of clothes in some form or fashion while every woman is most likely wearing a big dress because that was the style of the time. It all comes down to context, and if you look through enough photographs of a particular time, you'll find it all.