You're making a lot of assumptions with that statement which are fundamentally false

1. People of all classes had access to ANY music
2. People of all classes had access to non-religious music
3. People of all classes had access to the same music

Until the advent of recorded music, music was very much a commodity of the upper classes. If you were a peasant in Europe, and you or your family did not have the capacity to play an instrument or sing, you would likely not hear music in your life at all, unless you heard it in church.

And if you did have the capacity to play an instrument or sing, it would be folk songs that would not be heard/performed/sung by or to the upper classes.

And the upper classes would have the capacity to hire musicians and composers to write and perform for them, and there's no reason these musicians would ever perform for lower classes who weren't able to afford it.

So many of the questions about music during history come with the modern bias that music is this ubiquitous commodity. Today, we hear music in every bathroom, bus station, grocery store, and elevator. It's as common as wallpaper, and we take it for granted. We assume music is this universal constant - something everyone had access to always, the way we do now, whereas the exact opposite was the case. Unless you or your family or village were able to create and perform your own music, you could conceivably go your entire life without hearing music at all outside of church.