I know that fashion back then for women mainly consisted of extremely large and over-the-top gowns, but did rich women back then really wear them as casual wear? We’re you considered poor if you were caught not wearing those kind of dresses?
You will be more likely to get an answer if you specify a place!
"Large" and "over-the-top" are in the eye of the beholder - they're value judgements. That is to say, gowns were very common garments, but I can't say whether you personally would consider informal gowns over-the-top as well as formal ones, or indeed whether the portraits informing your opinion are formal or informal.
My flair profile has a section on eighteenth-century fashion with links to a number of my past answers on women's dress in this period, including how it varied by time and class.