I have to limit myself to the antebellum South, since I don't really study afterwards much. That said, the answer is the ever-popular "it depends and it's complicated." The free black population leaned somewhat male and free blacks were at least slightly scandalous in most of the South, so the most likely relationship between a white man and a black woman would be between enslaver and enslaved.

Fair warning, this gets ugly really fast.

It was accepted in the antebellum South that white men were going to prey upon black women, to the point where there's an established trade in "fancy" slaves bought chiefly for their sex appeal. They would be advertised as such, often with comments about how light their skin is. One particularly vile occasion involved Nathan Bedford-Forrest trying to sell an enslaved woman he claimed was the daughter of Frederick Douglass this way. Supposedly he got in touch with Douglass and offered him first right of refusal or something like that. Though I'm not a Douglass scholar, I haven't seen anybody who thinks that poor woman really was his kid.

This isn't something that Southerners necessarily trumpet as a virtue of their system, but at least some of them went that far. In defending the project to expand slavery from Missouri into Kansas, Benjamin Franklin Stringfellow wrote

Negro slavery has a further effect on the character of the white woman, which should commend the institution to all who love the white race more than they do the negro. It is a shield to the virtue of the white woman.

So long as man is lewd, woman will be his victim. Those who are forced to occupy a menial position have ever been, will ever be most tempted, least protected: this is one of the evils of slavery; it attends all who are in that abject condition from the beautiful Circassian to the sable daughter of Africa. While we admit the selfishness of the sentiment, we are free to declare, we love the white woman so much, we would save her even at the sacrifice of the negro: would throw around her every shield, keep her out of the way of temptation.

In other words, raping slave women isn't quite ideal...but the alternative is to afflict white women. That's fairly close to the consensus, though Stringfellow is unusually blunt about it. The use of female slaves for white male sexual gratification is normal, if not celebrated.

One might have consent, and even have genuinely mutual affection, but within the bonds of slavery those don't really decide things. The enslaver has the power to destroy the enslaved, as a practical matter. Refusal could mean being whipped to death, with or without sexual violence, and all manner of less severe but still grotesque physical punishment for oneself and one's family. Even a pair of people who really are into one another have that hanging between them.

Whether purpose-bought fancy slaves or "concubines" chosen from the local quarters, enslaved women are generally available to enslavers and their male relations. A relationship, so far as it goes, might be fleeting or go on for years. For the most part, a sense of not asking many questions seems to have prevailed. An enslaver's multiracial children might be a source of local gossip, but as long as he maintained some level of discretion not much was likely to happen either way.

Discretion usually meant not treating the enslaved woman as a wife or anything close equivalent. That would shame any actual wife and might outrage the family of a deceased spouse. But you could set her up with nicer quarters (maybe in the big house) and make assignations as you liked. Buying good clothes and other things like that for her wouldn't cause too much uproar.

But there are limits. Richard Johnson, Martin Van Buren's Vice-President, fell one electoral vote short of winning his office and had to be elected by the Senate because he was in an open, longterm relationship with an enslaved woman who was one-eighth black. She had fairly extensive privileges, including managing Johnson's affairs while he was away. It doesn't appear to have upset his neighbors too much, despite being fairly unusual, since they sent him to the House of Representatives a few times. Kentucky made him a Senator for one term, but his common law wife was part of the reason he didn't get a second.

But Johnson's is an unusual situation. Consider the case of Celia and her enslaver, Robert Newsom: He lived in Callaway County, Missouri, and did pretty well for himself. By 1850 he owned eight hundred acres and five slaves (four adult men and a boy). His wife died in 1849 but he had two adult daughters (one probably a widow) and grandchildren on hand. Two adults sons lived nearby. Newsom was sixty in 1850.

Celia was a new addition in 1850. We only know that she came from the county immediately to the north. She was fourteen-ish. (Slave ages can be hard to pin down.) Newsom had a daughter who was likely to run his household for the foreseeable future. He could probably have remarried, either to someone of his own middling class or a little bit less well-off, without too much trouble. We don't know why he chose to buy a teenage girl instead of find a wife for another go, but he did and Celia was pretty clearly meant for sexual use.

We don't know that Newsom's daughters thought about all of this. They could have been in denial, but they shared a relatively small space. There was probably a mix of anger and resentment, which they may have taken out on Celia. His sons might have hoped to share with the old man. Such things happened in some other households. Even if not, they probably wouldn't have voiced objections if they had them.

Celia gave birth to two children over the next five years, probably both courtesy of Newsom. He built her her own cottage, of brick, about fifty yards from the farm house. That might have been a reward for her, whether she saw it that way or not, because most slave quarters were far more modest. It also kept her a little bit away from his male slaves.

Sometime toward the end of the five years, an enslaved man named George started staying with Celia. This went on for a while and Newsom kept calling on her. In the late winter of 1854-5, probably in February or March, Celia found out she was pregnant for the third time. She didn't know by whom. She told George and George demanded she break it off with Newsom, or they were done. This is a lose-lose for everyone. Celia can't break it off because slavery. George can't do anything about it for the same reason. If either of them try, they could face serious retaliation. If Celia resisted Newsom by force, he could straight up kill her or take it out on her two kids. If George tried to get in the way, he might be beaten or sold off.

What are you going to do? Celia tried going to Newsom's daughters pleading pregnancy and sickness. It was a long shot, but George apparently meant a lot to her. It's likely that they did nothing, even if they believed her. They too were members of the slaveholding class and economically dependent on Robert. On or about July 23, she tried Newsom himself. He told her that he wasn't stopping and she had best be ready for him that night. Celia made some kind of physical threat, which we know from hindsight that he didn't take seriously.

Celia went back to her cabin and got a large stick. She put it in a corner, where it remained when Newsom showed up around ten at night. Her two kids were with her, probably asleep. They had a talk that ended with her cracking him over the head twice. He didn't survive the second blow. She burned the body in her fireplace and in the morning got Newsom's grandson to scatter the ashes in exchange for two dozen walnuts.

The only really sensational part of Celia's experience is the murder and resulting trial. (She was hanged.) Except for those, her experience appears to have been typical of what most enslaved women in her situation experienced.

Sources

What Hath God Wrought by Daniel Howe discusses Johnson's situation.

Celia, a Slave by Melton McLaurin is the grim microhistory of Celia's plight, set against the backdrop of Missouri and Kansas slavery politics.

Negro-Slavery, No Evil. by Benjamin Stringfellow (I have a PDF of it; there used to be a website version but it's gone down.)

Soul by Soul by Walter Johnson is largely a history of the slave trade focused on New Orleans and discusses fancy slaves in some detail.