It's such a complicated situation yet i can't seem to understand exactly why did they choose to essentially seek genocide as the main option. Why not just drive them away? it seems like there were so many other options yet they specifically chose this and most importantly many serbian civilians, government officials and military officers supported this, so it must be for a reason right? as irrational or farfetched as it might be.
Some seemed to try to justify it through the ottoman opression and things of that sort but it had been almost two hundred years since their independence by then and it doesn't seem to be a widespread narrative anyway. What exactly was the serbian nationalists' motivations to commit such acts? In the spirit of trying to understand all sides, feel free to correct any of the things i may have gotten wrong about the conflict nonetheless.
It's a complicated issue. I hope I can clear it up, as someone who had distant family members fighting on the Bosnian Serb side. And again, this is from the Serbian point of view, why was there such an incredible hate towards Bosniaks.
First: There wasn't any coherent ideology to genocide Bosniaks, but they weren't welcome in serbian controlled areas (read they wanted to get rid of them), so there were many cases of Serbian army completely banishing them and killing war prisoners, including women and old people. Politically, they were seen as enemies and separatists from the Yugoslav state and having to deal with less of them (by shooting PoWs and indiscriminately bombing Bosniak held towns) was an adventage both for the army and the politicians (evil reality of the war). Big percentage of crimes was commited by paramilitaries formed from prisoners (something like penal battalions) and volunteers who had their own hatred against bosniaks, croats, ideas of great Serbia etc. In the end of the war, when it became clear that Bosnian Serbs are going to lose without the help of Serbia, Mladić (who was mentally unstable at that time) ordered/or didn't object the Srebrenica massacre, again because it meant less enemy fighters, and the destruction of Naser Oric's troops. There were some RS politicians like Biljana Plavšić who saw Bosniaks as genetic waste though and that probably had some influence.
Second: If you want to understand why so many Serbs (from Bosnia and Croatia) hated Bosniaks and Croats, you have to get back to ww2. Majority of Bosniaks and Croats were in the ustashe ranks (fascists), while huge majority of Serbs were fighting as anti-fascist partisans, for their own survival. There was an industrial scale genocide against Serbs during ww2 in Bosnia and Croatia, so that made a huge mark on Serbian way of thinking and a huge stain on the inter-ethnic relationships. That, and the fact that Serbs felt they invested so much in Yugoslavia, that they didn't want to see it go down. When the war started, lots and lots of young Serbs from Bosnia couldn't wait to get the revenge for the ww2 crimes commited against their parents/grandparents and nation. That's why you have such disparity between war crimes (I think 70-80% of civilians killed in that war were Muslims), Serbs just hated Bosniaks more at that time and the army didn't care for Bosniak civilan casualties in lots of cases.
There aren't any good books or sources for this matter, it's just something "don't ask, don't tell" in the Bosnian Serb community and their way of thinking.
Edit: Keep in mind that Serbia and Republika Srpska (Bosnian Serbs) were two different entities, and Serbia didn't officially participate in Bosnian war after the 1992., but it did help Bosnian Serbs with arms and equipment.