Well, a couple of reasons.

1.) Banjo has a very piercing, high-volume sound even when acoustic, so a lot of the things that drive the electric amplification of the guitar (like needing guitars to be audible in a big-band setting) aren't quite as large of an imperative with the banjo.

2.) As you alluded to, the banjo was wielded as a racist prop within the minstrel tradition. This had at least three crucial effects. A.) It increased the number of White banjo players (though a lot of the White banjo tradition actually came about as a result of free cultural exchange amongst poor White and Black populations in the appalachian region post civil-war). B.) It drove the Black community away from the instrument, since they now wanted to be distanced from the way they were being portrayed by minstrel shows [SEE NOTE BELOW]. And C.) the banjo emerged as a comedic instrument, something we still see in, like, Steve Martin. And that is important because many early country barn shows and radio programs were a lot like variety shows with bandmembers engaging in comedic skits between songs. So the banjo's emerging comedic associations were both important to that function, and made it less desirable to angst-ridden baby boomers wanting to create "serious" music.

3.) The development and growing popularity of rock and roll throughout the 50s and early 60s was actually embraced by a large portion of the country community, and as a result, instruments like the banjo and fiddle (as well as dobro and other instruments), began to be de-emphasized in favor of more rock-and-roll type ensembles. But not all fans like this, many wanted a return to a "purer" acoustic sound: and this group basically served as the audience around which Bluegrass developed. So basically, country "purists" found these instruments so desirable precisely because they weren't all that common in pop and rock, and so they sought to define country's very identity through the use of those instruments. Bluegrass was essentially the result of that.

But of course, crossovers did happen. Earl Scruggs, the foundational bluegrass banjo player, famously moved pretty far away from the Bluegrass genre when he split with Lester Flatt. And of course you have newgrass bajoists like Bela Fleck who take the instrument into new genres all the time. But once the Folk Revival and Bluegrass claimed the banjo as a core feature of their respective genres, it was hard to shake those. There was a growing community of listeners who wanted the banjo to be central to their music precisely because it was not central to pop and rock music. And perhaps as a result, those outside of those communities increasingly regarded the banjo as this alien, antiquated instrument suitable mostly for uncultured country bumpkins. "The good ol' boys" were the ones that played banjo -- this is the instrument of freakin' "Rocky Top!" And that is a huge turn-off for rock and pop musicians, but very much a turn-on for country musicians!

For more on the banjo's history, especially its transformation from a Black-associated to a White-associated instrument, see Celia Conway's African Banjo Echoes in Appalachia. For information on the development of Bluegrass as a kind of "pure" country genre against modern rock- and pop-inflienced genres, see Neil Rosenberg's Bluegrass: A History.

[NOTE] I would be remiss, here, if I didn't bring up the notable number of women of color who have recently powerfully reclaimed the Blackness of the banjo. Give a listen to the supergroup Our Native Daughrers and its individual members (Rhiannon Giddens, Amythyst Kiah, etc.). It's fantastic stuff!