I've read that the language, Serbo-Croat, is spoken by all three groups and that it's mainly religion that separates them. Were they once one culture or ethnicity? Or was that not really a concept until they were separate?
Weeeell, define culture.
There are certain similarities in that they are all Slavic, and in roughly the same area, and have many similar traditions, but that goes for cultures which are outside of them, for example, slovaks.
Using Slovakia as an example, modern day consensus is that slovak is a single culture, but the reality is that the official slovak language is an artificial construct - there are three distinct groups of regional variations, and the differences aren't small ones, e.g. word for potato is very different in all of them (krumpla, švábka, bandurka, going west to east). While slovak speakers can understand these dialects to a great degree, we can also understand polish, croatian, serbian and so on in the same way.
Historically speaking, slavic tribes in this area were numerous and their traditions differed slightly from one another. old slovakian Perún is a god similar to Lithuanian Perkunas, but not quite the same. What you have here is, in essence, the exact reverse process than the one in your question - Serbo-Croat folks began as a myriad of tribes with differing customs and over time became somewhat more unified into several larger cultural blocks.
Oh, and remember what I said about slovak language being artificial? Nationalism only became a thing after French revolution, and especially before WW2 in Austro-Hungarian empire. Most of what you know today as modern slavic nation states were artificially divided at this point in time - hell, Slovakia has a mountain in its coat of arms that isn't in its territory because someone's hand slipped over the map and no one noticed.
This means that pre-modern people didn't care that much about their ethnicity/culture, a medieval croat was first and foremost a christian or a jew, and then a subject of Hungarian crown.
There's a lot more to this topic, considering that Balkans were invaded by just about everybody (including conversion attempts, sometimes with force), and ended up a tremendous chaotic mess, but unless you have more specific questions, that is a topic for several books.
I can't believe I put WHERE instead of WERE. So embarrassing.