Some research makes the argument that it was perhaps not as unheard of as we assume. Martha Hodes wrote a great book in the mid 1990s called White Women, Black Men. She argues that interracial relationships between white women and black men were far more common than is typically believed.

Her work focuses more on lower class women, but the man who was the subject of my master's thesis was the son of a wealthy white woman and a slave. He was born in 1789 in Georgia, and was raised by his mother until he was apprenticed to a plaster worker at 11. His mother had married at some point during Solomon's childhood, so a bastard child of mixed race was not necessarily enough to prevent a woman with sufficient financial means from making a respectable marriage later on.

If you're interested in the cultural machinations that prevented white Southern women from having families with slaves, I would recommend Bertram Wyatt Brown's Honor and Violence in the Old South. In chapter five he discusses the place of white women in the context of a Southern patriarchy. The cult of domesticity was pervasive throughout the United States, and Wyatt Brown argues that white Southern women were seen as indicators of their husbands' honor. Virtuous, dutiful women were as much a part of the carefully cultivated Southern ecosystem as their brash, martial, and honor-bound male counterparts.

I hope this is helpful. I would expound more if I weren't on mobile. I'd be happy to answer any other questions you have. This is exactly my cup of tea, and honestly an area that I believe more scholars will begin to research in the next few years. It's really fertile ground!