Could you narrow this question down to a century and region?

What was music like before that time, whether religious or pagan?

We don't really know much about actual music from antiquity. The Romans absorbed Greek theory, and they had a form of notation. We don't have many examples of it, and it was not the most explicit notation, so we can only speculate. To make matters worse, that notation died. By the 7th century Isidore of Seville wrote something along the lines of "unless sounds are held by the memory of man, they perish, because they cannot be written down."

I know there was music happening outside the Catholic Church, but much of the European world's music centered around prayers during mass.

In history courses you are told about religious music because we have evidence for that, and not for other kinds of music. There was A LOT of effort to standardize, distribute, and preserve chant. No such thing was done for other kinds of music until later.

If Gregorian chanting was the beginning of Western music,

Well, it is the oldest thing we can directly link to the Western canon (because that's when modern notation was created). The ancient Greeks produced a lot of theory and that theory was in some or another way kind of alive (they really were trying to keep using it even when it was meant for a very different music). You find books dealing with some form of theory from antiquity even after the so-called Renaissance.

if other more advanced music was going on before that

Advanced? Advanced kind of implies "progress," as if there was some direction and/or a final objective. It is not very useful to judge music from the past as primitive or "less developed." It's all relative...

such a simple and primitive form of music like chanting

ಠ_ಠ