Or was/is it just a catch-all that was used by Christians to mean “non-Christian?”
There's certainly more to be said, buy it was historically a catchall for people who didn't worship the same God as Christians, which I describe in more detail in this answer with Zoroastrianism as a specific example.