A portrait from around 1900 on the family farm has a cabin in the background that my relative will quaintly affirm to be servants' quarters. However, even though we know that slaves existed, family lore is not forthcoming on any details.
I just watched "12 Years a Slave" yesterday, and other than making me unable to sleep last night, it also made masters sleeping with their prettier female slaves seem very common. So I wonder how likely it is that any of my esteemed Southern ancestors did that.
Also, how the eff did slaves cope with having families that could be torn apart at a master's whim? (I'm thinking of Eliza here) Seriously, WTF?!
It'd ultimately depend on your relative in question. There were many different types of slave owners: Some treated their slaves kind of like "pets," so to speak; they were definitely unequal to the rest of the house, but they were still considered beloved members of the family, who wouldn't dream of trading them to another person and genuinely tried to see to their personal needs if they could (This was most common with house slaves, especially nannies who raised the children). There were owners who viewed them neutrally and entirely as a business investment. These owners would not necessarily be cruel, and could in fact be quite benevolent and generous with freedoms on the estate when it suited them, but ultimately only cared about the bottom line and productivity. Some owners treated their slaves as the stereotypical, whip-cracking, crooked-teethed slave driver we've been raised to see. They were cruel and uncaring, and would do whatever they wanted with their property as they pleased.
The only way you can find that out is to learn more about your relative, and what type of estate he owned and how many slaves he had. A private investigator would be a great help to you.