Hoo boy. This is a huge can of worms, but I'll give it a shot.

By its very nature, quantifying the number of slave owners who had sex with their slaves is very difficult. We do know that there were not a lot of stigmas attached to these sorts of relationships however. It was not exactly something that you would talk about at a fancy dinner, but it was not exactly something that many slave owners hid either. It was, in many ways, an open secret. There's a fantastic article by Ed Baptist that shows that there was an entire subsection of the domestic slave trade dedicated to the sale of attractive young slave women, who would then be used for sexual pleasure by their slave owner. While the slave owners might have been reticent to discuss their sexual doings, the slave dealers by and large were not. Thus, the fact that this particular section of the slave trade existed is a pretty suggestive case for the prevalence of sex between masters and slaves. As slaves were being moved from one part of the country to another it was not uncommon for men to take a "female from the drove to lodge with him, as was the common practice. (Ed Baptist, The Half Has Never Been Told. There are some issues with this book so be moderately careful about using it as a source, even though I just did.)

At the time, common beliefs held that slave women were naturally promiscuous, so even though they were slaves, in the slave owners eyes they "wanted it." This belief lasted an unfortunately long time and led to a lot of sexual encounters that we would now classify as rape. But, that gets at the heart of the issue. Was it actually rape or was it consensual. There are some who would say that by the very nature of the slave/master relationship, consent was impossible and therefore all sexual encounters between a master and a slave was rape. Nevertheless, there were some advantages to being the master's mistress, so to speak, such as a "lighter" work load and the possibility of obtaining material goods, or even freedom. Unfortunately, it is very difficult to get at the emotional attachment that a master and a slave might have for each other. There area few examples of slave women who seemed genuinely attached to their masters and vice versa, but I don't know of any way to judge if that was the norm or not.

As far as social repercussions, go there were few. Southern slave owners generally considered it their due and if they thought about it at all, then they probably would have thought that the slave was willing. I've even read the diary of a plantation wife who was glad that her husband has a slave mistress because he was "whitening" the black race. Not to mention that slaves were expensive, so getting your slave pregnant was a way to increase your capital if one was a slave owner.

For more information you might want to check out this post by /u/sowser.

Sources: Ed Baptist, "Cuffy, Fancy Maids, and One Eyed Men". Baptist, The Half Has Never Been Told, Camp, Closer to Freedom, Clark, Strange History of the American Quadroon, White, Ar'n't I a Woman.