Part 1

The very short answer to that is history. The slightly longer answer is that it's complicated. And the answer extensive enough to capture the whole situation would fill a book or two at least.

The first complicating factor here starts right with the premise. I'll also get to the difficulty of what and who is an ethnicity in this post but the first thing that jumped out at me is that you use the term "Bosnian". The official demonym for citizens of the country Bosnia and Herzegovina (BiH) is "Bosnian-Herzegovinan", while the generally held break-down of the composition of inhabitants of BiH is 50% Bosniaks, 30% Serbs, 15% Croats and the rest "other". "Bosniaks" is traditionally a term used to describe the Slavic inhabitants of the region who converted to Islam during the reign of the Ottoman Empire and are also often colloquially referred to as "Bosnian Muslims" or during the first half of socialist Yugoslavia's existence as simply "Muslims" (despite the fact that a majority of Albanians are also Muslims (58% by latest estimates).

The term "Bosniaks" is not uncontroversial however because certain Serbs and Croats claim that there is no Bosniak ethnic group but that Bosnian Muslims are simply Croats or Serbs of the Muslim faith, not their own ethnic group. The term has even inspired some controversy among Bosnian Muslims themselves where there are people who despite Bosnian Muslims having suffered genocide during the Yugoslav wars reject the label of being their own ethnic group preferring to think themselves as Bosnians and including in that definition also Bosnian of Orthodox and Catholic faith, usually referred to as Croats and Serbs in Bosnia. At the same time, there are Muslims living in Croatia and Serbia who will tell you that they indeed are Serbs and Croats of the Muslim faith rather than Bosniaks.

So, who is Bosnian, who is Bosniak, who is Croat or Serb of Muslim faith? For that matter, who is Croat, Serb or even Slovenian for that matter? Take Friedrich Engels who dismissively referred to the Slovenians as a "people without history" at the same time as German authors compiling an atlas of all the peoples of the Habsburg Monarchy said of the Slovenians that they were Germans who spoke a Slavic language.

This short discussion of just a few of the difficulties here in even pining down who the ethnic groups are is really related to the inherit problem of what ethnicity is and, closely related to that, a matter of history.

Ethnicity seems at first glance like a really and obvious concept. I mean, here you have people who share a history, follow this religion, having this specific culture, speaking this specific language. Next to them, there are other people who share a different history, who follow this different religion, this different specific culture, and speak this different language. It would make sense to say that these people are two different ethnic groups.

But where do we draw the line in that? Bavarians and Prussian have a different history, follow different religions (Catholicism and Protestantism resp.), have an arguably different culture (Lederhosen vs. Prussian Militarism), and arguably speak languages that are about as different from another as Croatian and Bosnian. And yet, only the fringiest of already fringe Bavarian separatists would claim that Bavarians are a different ethnicity than Prussians. Today we consider them ethnically German, a category into which both Austrians and German-speaking Swiss are also lumped in despite even starker differences both language, culture, and history-wise.

Or take the French. The territory of what today comprises France has historically been home to the following so-called ethno-linguistic groups: Alemannic Germans, Arpitans, Basques, Bretons, Catalans, Corsicans, Flemings, Occitans, and Franciens. Of those groups, we more than willingly acknowledge the Germans, Basques, Catalans, Flemings (as Dutch) and maybe even the Corsicans as their own ethnic groups while Bretons, Occitans, and Franciens are all happily claimed (by non-Bretons etc.) as generally French.

What all this shows is that ethnicity is something that is not eternal or natural but something that needs to be created, asserted, and acknowledged and that is very much subject to historical change and thus contingent on historical processes. For France, Eugen Weber has described this process as "making peasants into Frenchmen" and what he means by that is that through a historical political and social process, the people living in the state of France were made French, meaning that they started understanding themselves as one community of people, possessing a shared culture and history; they started seeing themselves as French and so the idea of a French ethnicity came into being. The process of how this worked is manifold but it for example encompassed the – sometimes forcible – spread of the what had been defined as the French language. As late as 1863 at least a quarter of the people living in France did not speak French, rather they spoke Occitan, Corsican, Breton and a variety of different patois from all of these. The French language was spread as a project of national elites through school education and also through a variety of social changes that simply came with having a modern state: Army service, the spread of modern bureaucracy, the growth of the central state – all these contributed to the spread of the French language and to the spread of the people living in France seeing themselves and being seen as French.

Elsewhere it is not different and because this is a very political project that entails the potential for a lot of power, it sometimes result in conflict: Germany was unified into a German nation by Prussia and did not include Austria because of the relationship between the Habsburg and the Hohenzoller houses. And that is just the most obvious example. Norwegians and Swedes, the Dutch and Flemish, the long and storied history of many different Italies and so on and so forth.

This is essentially what inspired Rogers Brubaker to formulate the concept of groupism. Brubaker writes that

Ethnicity, race, and nation should be conceptualized not as substances or things or entities or collective individuals – as the imagery of discrete, concrete, tangible, bounded and enduring "groups" encourages us to do – but rather in relational, processual, dynamic, and disaggregated terms. This means thinking of ethnicity, race, and nation not in terms of substantial groups or entities but in terms of practical categories, cultural idoms, cognitive schemas, discursive frames, organized routines, institutional forms, political projects and cognitive events. It means thinking of ethnicization, racilization and nationalization as political, social, cultural and psychological processes.

and therefore that for example

We need to break, for example, with the seemingly obvious and uncontroversial point that ethnic conflict involves conflict between ethnic groups. I want to suggest that ethnic conflict – or what might be better called ethnicized or ethnically framed conflict – need not, and should not, be understood as conflict between ethnic groups [emphasize original].

Rather, we need to understand such categories as ethnic or other groupist terms as something invoked and constructed by historical actors. It is these actors who cast ethnic, racial or national groups as the protagonists of conflict, of struggle. In fact, these categories, while essential to the actors casting them, referencing them, are in themselves a construct, a performance.

So, while we can now say that ethnicity is a historic process and performance, that still leaves the question about the specifics of this in the Balkan region and why, e.g., today we encounter Croatians and Serbs and not, say, a unified Yugoslavia where ethnicity is understood and performed the same way as in a unified Germany.