In Ancient Rome even if it wasn't considered illegal or even particularly immoral, would it have been socially unacceptable to openly talk about the fact you had sex with your slaves?

I think, as it always is with speaking about sex, it depended on the social circle. Vern L. Burroughs' book The Subordinated Sex: A History of Attitudes Towards Women has a chapter titled "The 'Rise' of Women and the Fall of Rome." In that chapter, there's a discussion of women as slaves having to do with their status as mistresses and the hope of matrimony. I'll give you the paragraph here:

"Even under Augustus the slave woman who attracted her master had little choice, if he was married, but to become his mistress, or one of his many mistresses. Most slaveholders had slave women, and some of the women even earned their freedom by serving their masters. Cato, the guardian of public morals mentioned earlier, was greatly surprised to find that his son and daughter-in-law were irritated by the fact that he had kept a slave woman as a mistress while his wive was alive. This, after all, was a matter of custom, not of morals. [...] Concubines were found at all levels of the social ladder. For example, the emperors Vespasian, Marcus Aurelius, and Antonius Pius all took concubines after their wives has died. Marcus Aurelius claimed he did so because he did not wish his children to suffer from a stepmother, although in his meditations he had written that he was most happy he had managed to avoid overlong contact with his grandfather's concubine."

In this passage, both Cato and Marcus Aurelius seem to not only have talked about their relations with slaves, but included it in their respected writings. CAto's children even expressed their disapproval with Cato.

TL;DR: Gossip is gossip, no matter what the century.