Incredibly broad question, but I'm completely clueless on this.
"Classical music" is a musical tradition. It has its roots in medieval religious music and encompasses a lot of different things.
This music has been associated with the rich for a reason. Look at who were the employers of the "great masters" (churches in well-doing cities, rich people), look at where that music was played (not super poor tiny churches in the middle of nowhere, not at the taverns or the poor neighbourhoods), look at the dedications in their works (to count something, to princess this other... how many works are dedicated to poor people?)
Now, this music was not so separated from "normal everyday music." Religious services used music in this tradition, and everyday music shared elements with it. In our days classical music is very different from other forms of music people experience every day.
Was classical music the music of the people? Not really.
A lot of people could listen to the great music performed at Notre Dame Cathedral, but if you lived in the middle of nowhere your church (chapelle or whatever) would probably not have that kind of musical life.
Beethoven's quartets were published and people could buy the sheet music but it was not played in the pub (and to be honest your average amateur would struggle to play them), not every little church was able to have performances of Mozart's masses, not every person was able to pay the admission to the opera (and that's when operas were performed in public venues and people could just buy tickets, opera started as music for private events of the super rich).
Chopin's waltzes were not music for dancing, but were kind of social. That music was played at the soirées of very well doing people, Chopin would not be playing for poor people. The waltzes of Johann Strauss II are in the same musical tradition (he was born in a family of classical musicians) and were closer to being the music of the people, it was part of social events. I think these events were for middle class people and above.
The music of brass bands is part of the classical tradition. This music did make it closer to the public, because it was present in public celebrations, the military and so on. We know of things similar to these bands playing in public events in towns since the Renaissance (and I think the "Middle Ages").
It's not like only the "great masters" composed in the classical tradition. You do find a lot of music composed in small cities by unknown musicians (even amateurs some times) and good music is frequently being found in small churches around the world (composed for services in the past). But there are differences: it's expensive to train musicians, to build the places where they play (a lot of this music was meant to be played indoors), talent would go where the opportunities could be found... Quality and quantity (among other things) do vary.
So, classical music covers a lot of things. Some parts of it were closer to being "for everybody," but this music has been mostly the music of the rich. That doesn't mean poor people had no contact at all with some form of it.
The luxurious stigma you mention is a modern thing. Classical music is seen in a different way now because it is VERY different from everyday music, the strange rituals that surround it were created in the 19th century. Going to the opera was a completely different thing back in the 18th century.
hi! there's lots of room for the music historians to contribute more here, but meanwhile, you may be interested in some earlier posts
This is a broad question, largely because it's hard to know what you mean by "Classical" music.
The kind of music that most immediately comes to mind when most people think of classical music is that of the Classical era (think Mozart and Haydn) - light, airy, harmonically simple, balanced.
Today, the term classical music is synonymous with the slightly more accurate term "Western art music", which can encompass music from as far back as the Medieval and Renaissance eras. Especially if we are allowing our definition of classical music to include music from those eras, then the answer to your question is a resounding no - at various times throughout history, "art" music has been used for partying/revelry, as well as an integral part of worship services. Those are two scenarios which were more than accessible for the common person.
With the advent of printed music, scores (sheet music) and instructional manuals were able to be distributed more widely, which led to widespread interest in amateur music-making by the middle classes in Europe. Much of the chamber music written by composers such as J.S. Bach would have had such an amateur enthusiast in mind as its target audience.
The Romantic era (typically considered as beginning somewhere around 1810-1820 and ending around the start of the 20th century) is where a lot of our preconceptions relating to classical music's "stuffiness" began. It was at this time that concerts began to be much less raucous affairs (before, it was par for the course to boo/hiss, cheer, or otherwise be distracted during concerts). In the Romantic era, concert attire became much more formalized, the audience was expected to pay rapt, silent attention to the performer, and the musician-poet's genius was venerated above all else.
The short answer to your question is (if I am interpreting your question correctly) that the luxurious stigma you speak of is largely an effect from the musical customs that came about during the Romantic era (late 19th century).