I've come across some claims that there was a kind of ritualized public rape of male slaves used as punishment, referred to as "breaking the buck," but I don't know whether the sources I've come across are reliable.
Would a slave owner in the antebellum southern US who preferred men be regarded as particularly scandalous, if his activities became known to the local (free/white) community? Or would it be more like a slave owner and slave women -- i.e. it's not something you would publicly brag about, exactly, but if people found out you had such proclivities, it's not a big deal?
While it doesn't directly address your question, /u/sowser's answer is a particularly in-depth look at whether buck breaking existed (no) and why same-gender sexual exploitation of slaves is is probably not captured that well in the historical record.