It's funny, I've been thinking about this topic lately.

So first of all, yes, there's a huge difference. During the late eighteenth/early nineteenth centuries, while married women in the United States had very few rights in law separate from their husbands - due to the US incorporating the English common-law practice of coverture - they were not legally enslaved, and they had considerable social protections in comparison to Black women, particularly enslaved women.

I've written previously about spousal abuse during this period - to sum it up, while seventeenth-century New England colonies cared a lot about legislating morality, including relations between husband and wife, by the eighteenth century this was of far less interest to American authorities, and there was nothing seen as inherently wrong in either time period with a husband disciplining his wife as a father might a daughter. Among the working classes, men seen as overly abusive/unreasonable in their discipline could face social shaming through "rough music", demonstrations outside their houses that often involved pulling them out and embarrassing them; middle- and upper-class wives would instead rely on their own relations to step in privately to defend them. However, enslaved women had no protections at all, and the lengths to which their enslavers were socially permitted to "discipline" them amounted to torture - the law didn't care, hardly any White people did, and the other Black people who might have were completely unable to intervene. No White woman could be tied to a post by her husband, stripped, and whipped until she bled without some kind of retribution, but this was absolutely expected/feared by many enslaved woman who knew that nothing would stop it if their enslavers chose to do it. A White woman who wanted to divorce her husband for abuse faced significant barriers, in that she would have had to prove in court that her husband was extremely abusive and that he had deserted her or committed flagrant adultery - but of course enslaved women did not have any kind of "divorce" from their enslavers. White women's husbands could send away their children to school or to relatives without their wives' consent, but enslavers could and did sell the children of enslaved Black women so that the families were never united again. Marriage of the time certainly had some things in common with slavery, but slavery relegated women to a social death and non-personhood while marriage resulted in a more nuanced and slightly more equitable relationship.

However, it's vitally important to understand the nuances of that relationship. Driven by fictional representations, people today have a tendency to divide historical marriages into Bad Marriages (with abusive, much older husbands who dishonored their wives in various ways) and Good Marriages (loving relationships we can accept as romantic and stable). In reality, we have to understand that the lack of legal protections from abuse and particularly rape - it's not until fairly recently that sexual assault of a spouse even became considered a crime, because they were seen as owing their bodies to each other - meant that every Good Marriage had the potential to become a bad one, and that every Good Marriage was only good at the husband's discretion. While some White women of this period did critique the lack of rights they had in marriage, to the vast majority this was simply normal. The abuse of a wife was often seen as justified even by other women, and women who were abused frequently rested their accusations against their husbands on the basis of the abuse not being justified for themselves because they had been industrious and faithful wives.

This doesn't mean that we should then look at every single male/female relationship as a stereotypical Bad Marriage, but that we should be aware that inequality and love (or at least acceptance) can exist side-by-side. If you argue that enslaved women can never be said to have consented to sexual relationships with White men, you also ought to look at free women through the same lens. (I would continue this forward in time, but I'm trying to avoid soapboxing.) And vice versa, if you can conceive of married women in loving relationships despite their lack of power, you should also be able to picture that despite the inhumane bonds of slavery, relationships between the enslaved and their enslavers could be more complex than that between Patsey and Epps in Twelve Years a Slave. This is what Annette Gordon-Reed (whose work I would strongly suggest you read if you are interested in this topic) argues with regard to Thomas Jefferson and Sally Hemings, as well as other relationships that were unremarkable in their day but now are problematic. Gordon-Reed's point is, basically, that while the power imbalances make true consent impossible, assuming that every single relationship between a Black woman and White man, historically, was one of victimization infantilizes the women involved. From The Hemingses of Monticello:

To ensure that everyone gets the vital message that the rape of black women was endemic to slavery, the no-possible-consent rule says that whether Jefferson used force or charm on Hemings is of no great moment. Social history trumps individual biography. But one can safely say that for Hemings, who lived her life as a person, not a statistic, the difference between being forced, physically or psychologically, by a man and being charmed by him would have made all the difference in the world to her inner life, a thing that was and is, indeed, always of great moment. [...]

The profanity of slavery does not define the entirety of the lives of enslaved people so that everything any one of them ever did, felt, or thought—everyone they touched, every situation in which they were involved, every connection they made—was degraded. There is an inherent danger in automatically transforming women like Sally and Mary Hemings into something they themselves may never have thought they were in order to convey a message about the overall character of slavery—to treat these women as vehicles rather than as persons. No individual’s life, hopes, struggles, and dreams should be sacrificed to that instrumentalist end.

Gordon-Reed notes that enslaved women have been portrayed as race traitors if they accepted the attentions of White men, and even when they didn't, Black men in their lives sometimes took their sexual victimization as insults to themselves. We are still mired in the portrayal of women as either virtuous wives/mothers or ambitious whores, which results in trying to peel back the layers of assumed degradation and victimization coming off as moving a particular woman into the "wrong category", but the reality is more complex.

As far as why Thomas Jefferson ... it's because he was massively hypocritical. He said that slavery was wrong while being an owner of slaves, and while having some form of enduring sexual relationship with one of the women he owned. The fact that he kept it secret at the time shows that even he was aware that his relationship with Sally Hemings didn't quite fit with his beliefs and wouldn't reflect well on him. He's also been made out to be rather perfect, like the rest of the Founders, which is simply an unfair characterization that deserves to be challenged. It seems so obvious to me that I don't really know how to respond to that aspect of your question.