Basically, and Bosnians might not like me saying this, there is not a huge difference between Croats, Bosniaks, and Serbs ethnically speaking. Their languages are mutually intelligible and the main point of division is religion. Croats are Catholic, Serbs are Orthodox, and Bosniaks are Muslim. So the diversity is more about religion than anything.

For example, during the genocide, Serb soldiers would force captured men to prove they weren't circumcised, because that was really the only tangible difference between them and the Muslim Bosniaks.

As for why, Bosnia once had its own church when it was an independent kingdom in the middle ages. Bosnia was conquered by the Ottomans, and many of the church's adherents converted to Islam in order to receive preferential treatment from the Muslim empire. These people became the Bosniaks.

Borders in the Balkans were constantly changing, and often ill-defined, so a Serb could move one village over and find themselves in a different country. In the middle ages, Serbian, Croatian, and Bosnian kingdoms waxed and waned, and their citizens would spread out to conquered areas only for them to be conquered by another kingdom. Eventually the whole region was conquered by the Austro-Hungarians and the Ottomans, who drew borders based on the territory they controlled (Bosnia being the frontier between the two).

If you look at nationalist maps of "Greater Croatia" or "Greater Serbia", the spread of these people looks a bit more straightforward, corresponding more to the borders of medieval kingdoms than to modern borders.

Source: Blood and Vengeance by Chuck Sudetic