Brutus was the son of one of Caesar's most well attested mistresses, Servilia. After the assassination Brutus and the other assassins met at Servilia's house, so clearly there was some kind of relationship between mother and son at that point. Then again, she survived the purges of the second triumvirate, so clearly Caesar's successors didn't view her as a threat, which suggests she wasn't too involved. I'm not sure about the rest of them because in the absence of overwhelming piles of evidence (as in the cases of Servilia and Cleopatra) it's hard to parse out actual events from racy gossip and slander. Caesar obviously had serious detractors and if they were willing to murder him, most likely they were willing to spread untrue rumours about him as well. You didn't specify what standard of evidence you'd accept for "had sex with" so that makes it difficult.