The short and simple answer is Ottomans. The longer one goes something like this:

In Croatia, the Ottoman raids prior to conquest practically depopulated areas neighboring Bosnia. Most local Croats ran for safety beyond the reach of Ottoman Akinji's. Following the conquest Ottomans imported population, mostly muslims in that area, but following the second siege of Vienna and the loss of Ottoman teritories north of Sava/Danube, the locals, now mostly muslim were forced to leave, due to the convert, leave or die policy that reigned at the time, leaving the land desolate once more. At the same time that was happening, the Serbs were revolting in Kosovo, and following the pace settlement they were invited and settled to live on the Austrian border with Ottomans. The even is called the "Great migration of the Serbs" and about 40 000 Serb families participated in the event, settling in the mostly desolate areas from which Ottomans were forced, mostly Lika, Krajina and Slavonija.

For Bosnia it is important to understand that feudal system there lasted until 1922. Because of geographical position of Bosnia as the border of the empire, and due to several plagues that led to thousands of death and widescale depopulation, the landlords were forced to import serfs to toil the land. Those Serfs were mostly Orthodox peasants from Kosovo, Montenegro and Romania, the later of which would be assimilated into Serb culture due to Serb orthodox church. That is why if you overlay the geographic map of Bosnia over ethnic one you would notice that Serbs ocupy most of fertile land in Bosnia, granted to them following the agrarian reform of 1922.

So to sum it up, they had a large population, but had no land to call their own, and as such were moved all over the Balkans, leading to them being so spread out