So, the Serbian part of Bosnia and Herzegovina, the Republika Srpska, is the result of the violent disintegration of socialist Yugoslavia in the early 1990s. Like in Croatia with the Republika Srpska Krajina, the Republika Sprska in Bosnia was a proto-state declared by Serbian nationalist entrepreneurs during the process of independence and inspired by the nationalist policies that had emerged during this process and drew on previous nationalist tensions, mainly in the 1980s. Some of the territory encompassed by the Republika Srpska had historically encompassed territory where a majority of inhabitants had declared themselves Serbian, akin to how historically in Herzegovina, many people had declared themselves Croatian but going further back in history it is a bit more complicated than that.

What today constitutes Bosnia and Herzegovina (I go into the complicated reality of Bosnia and Herzegovina after the Dayton agreement in this answer) is for the most part the territory of the Bosnian Vilayet and Bosnian Eyalet, both administrative divisions in the Ottoman Empire. Like so many parts of the Ottoman Empire it was a heterogeneous territory when it came to religion, with a Muslim ruling and middle class and Catholic and Orthodox peasants. What set the Bosnian Eyalet/Vilayet apart was that on average, more peasants and merchants of the Slavic population there converted to Islam than in other territories. This had various reasons, from a greater lack of priests and presence of both Churches due to geography to it generally being harder in certain areas to earn enough money for taxes, which created an incentive to conversion because as a Muslim in the Ottoman Empire you didn't pay taxes. So, in this territory – and this is similar to Albania and Kosovo – it stands out that there were a greater number of Slavic-speaking, Muslim peasants in some areas, while in others peasants remained either Catholic or Orthodox.

In the 19th century, the ideological and practical formation of ethnicity/nationalities begins and they are often fused with revolts and protests over economic issues. For Bosnia two of these instances are important here: The revolt of Husein Gradaščević in the early 19th century and the Herzegovinan Revolt in 1875.

Gradaščević was a Slavic-speaking Muslim and lead a revolt against wide-sweeping administrative reforms of Sultan Mahmud II, called the Tanzimat reforms, that would have changed the economic situation in the country, including negative effects to Muslim landholders. He is important in as far as not only is he revered in Bosnia today as the first Bosniak (meaning Bosnian Muslim) hero but also because he used the Slavic-speaking Muslims in Bosnia as a power base against the often send there Muslim administrators of the Sublime Porte, thus in effect creating the idea that Slavic-speaking Muslims and those commonly referred to as Turks were different. It basically lead the groundwork for the understanding that Bosniaks were not merely Slavs who had converted to Islam but constituted their own nationality/ethnicity.

The same process was furthered by the Herzegovinan Revolt in 1875, only for the Catholic and Orthodox peasants who through this process came to understand themselves not merely as Slavs of Orthodox or Catholic faith but as Serbs and Croats. Hannes Grandits describes this in his article Violent social disintegration: a nation-building strategy in late Ottoman Herzegovina. in: Conflicting Loyalties in the Balkans. The Great Powers, the Ottoman Empire and Nation-Building, ed. by N. Clayer, H. Grandits and R. Pichler, London 2010, S. 110–134.

In it Grandits takes a closer look at how the Herzegovinan revolt that eventually lead into Serbo-Turkish and later into the Russo-Turkish wars of 1876–78 actually functioned on the ground. This uprising is often portrayed as both prove for ethnic conflict in the Balkans as well as for the rise of the nation against Ottoman rule in the area. The fact is though that it started out as neither. Rather, it started as something fairly common, a refusal by local political village leaders to pay taxes because the harvest had been bad in previous years.

Initially both sides were open to negotiations and a commission of a Pasha was send to the area to negotiate with the peasants. Within these negotiations however, a group of local nomadic bandits robbed an Ottoman caravan because they feared – rightly – that good relations between authorities and peasants would threaten the support they received from locals. This caravan attack lead to Ottoman reprisals and while things seemed to be winding again down after this, these reprisals caught the attention of young nationalists from Serbia and Montenegro that traveled to the area to enlist the help of aforementioned bandits to fan the flames of conflict. Suffice to say that despite the opposition of local village leaders, they were successful because the spiral of violence this relatively small group of people managed to lead to a homogenization of interests in the sense that local peasants were caught in between fronts and had to declare for one side for fear of retribution or generally violence.

Furthermore, because he saw the chance to gain territory, the Knajz (local ruler) of Montenegro "internationlized" the conflict by pleading for help from Garibaldi who sent him Italian volunteers to fight with the rebels against the Ottomans. It was really only this move which lead to an internal and external perception of the conflict as one that was "ethnic" or religious while most of the Herzegovians who actually fought on the ground had started this not to end Ottoman rule or "expel the Turk" but rather as rebellion simply to get central authorities to ease off with the taxes.

Through a process of violence and agitation, a relatively small group of what Roger Brubaker describes as nationalist entrepreneurs managed to homogenize the interests of the local population with their own politically goal.

In the aftermath of the revolt, the Bosnian Vilayet still continued to exist as such but was given to Austria-Hungary in the Berlin conference of 1878 as a sort of protectorate to administer. Austro-Hungarian rule in Bosnia is important because as a way to grapple with the newly formed nationalist identities of various people in Bosnia, the Austrians tried to establish some sort of overarching Bosnian identity or a people that spoke the same language and shared three equal religions. And while this fitted well with some segments of the population, in particular the Bosniaks, it remained ultimately not very successful in isolating the locals from the irredentism and nationalism of the various neighbors of Bosnia, specifically Serbia and Croatia.

This trend of trying to institutionalize politically Bosnia as a multi-ethnic territory was something that continued even in socialist Yugoslavia, where the territory was organized along the nacionalni ključ (the national key) which strove to enforce equal representation of Serbs, Croats and Bosniaks in Bosnia. And while this might have been done with an understandable sentiment behind it, because it it institutionalized the difference between these groups rather than emphasizing a shared identity as Bosnians, it lead to the situation in the early 1990s where both Serb nationalists and Croatian nationalists raised claims against Bosnia and under influence from Serb nationalists, the Republika Srpska was proclaimed.

In essence, because Bosnia was a territory created within the Ottoman Empire, it had always had a heterogeneous population, comprising Muslims, Orthodox and Catholics with the special contention that it had a higher number of Muslim peasants than elsewhere in the same region. Over the 19th century these religious differences morphed into newly created national and ethnic identities and thus Bosnia developed into a territory with Bosniaks, Croats and Serbs.