Previous generations talk fairly commonly about the days when it was uncommon to go out in public wearing the kind of clothing that we are now generally accustomed to seeing people wear in public, so I am curious to know at what point (if there was one singular point in our country's history) we started dressing more casually.
Casual daily dress was born in the early-mid 20th century on college campuses. Students rejected a more businesslike standard of dress and insisted on putting comfort and ease at the forefront of their considerations: sweaters instead of coats, no hats or gloves, etc. The style moved out into the mainstream in the late 1940s and 1950s, and became a marker of desirable middle-class-ness.
The social changes of the period (civil rights and feminism, expansion into the suburbs) married to technological changes (synthetic fibers, mass media, changes in department stores) were also incubators for the change. By the 1970s, the old order was completely gone, and clothing - like suits - that had previously been the middle-class standard of daily dress were relegated to formalwear.
Source: Dress Casual: How College Students Redefined American Style by Deirdre Clemente (2014)