The relationship between the Balkan nations is a lot more complex than just an identity based around being South Slavic.

The areas that joined Serbia and became Yugoslavia were occupied by colonial powers when they were handed to Serbia - Bosnia was part of the Austrian Empire along with Croatia. These areas had not had their independence for a long time and since that independence they had changed utterly - Muslims were now the majority in many Bosnian areas and Catholics were now the majority in Croatia.

Bulgaria had recently fought Serbia in the two Balkan wars and also claimed continuity with the old Bulgarian state, giving a much bigger sense of identity. They also recieved their independence much before the end of WW1.

Essentially it's a case of totally divergent histories, just being south slavic isn't enough at all. The same reason Poland didn't join Czechoslovakia, you know?