Basically medieval Bosnia had a somewhat unique population that had their unique church. Their leaders were catholic tho, due to the pressure from Vatican and strong neighbours. Medieval times had no national states so it was all mixed. Ottomans come and give privileges to the people that accept Islam. Many of the nobility do that to keep their privileges and many members of the bosnian church do it because their church wasn't so deep in the state. Additionally when Serbia gained independance, Bosnia and Sandzak didn't. Both of these accepted many refugees from the freed state as they weren't welcome there. That explains the big muslim population, the Croat and Serbs have been living there from medieval times but kept their religion since Ottomans somewhat allowed it. Initially Serbia wanted Bosnia to be part of it but the Austrian empire annexed it which led to high hostilities that contributed to WW1. After WW1 there were no Bosnians and they had to choose either Serb/Croat. Since WW2 broke out it didn't allow the integration to complete and lead to religios crimes etc which lead the muslims feel separated even more. Communists win WW2 and since they are for a federal state they make Bosnia a state and recognise Muslims as a nation. Then the breakup of Yugoslavia happens and the question of Bosnia comes up. Serbs and Croats wanted out but Muslims wanted a whole country, even though they wanted to leave Yugoslavia. After the war it was agreed to have a federal Bosnia as an independant state with two entities. However this makes all sides unhappy since Serbs/Croats still want out while Muslims want a more unified state where they run things - due to the population difference.