Question is inspired by the Spartacus TV show, and yes, I do realize it's hardly something to go off from.
In it, the owner of the gladiator gym would sometimes have sex with slaves while the wife was just around him. She would get pleasured by another female slave or be involved in a threesome with her husband and a female friend.
Her affair with a gladiator was a secret though, and she knew that her husband wouldn't accept it.
So, did Romans think that a husband or a wife sleeping with other women was fine but sex with a man that wasn't your husband was wrong?
Most of what my handy Oxford Classical Dictionary has to say concerns women. I'll quote the article below:
Roman tradition ascribed to fathers and husbands great severity in punishing illicit sexual behaviour by daughters or wives. Such misconduct was stuprum in married or unmarried women, an offence against chastity (pudicitia); adulterum described sexual intercourse between a married woman and a man other than her husband. Until the legislation of Augustus, regulation was chiefly in the hands of the family: adultery probably always justified divorce; a family council might advise the paterfamilias (husband or father in whose power the woman was) on this and other sanctions, possibly including execution. The immediate killing of adulterers/adulteresses taken in the act was defensible (morally and in court) but probably not legally prescribed. Other physical violence against the adulterer is a literary commonplace. Adultery in the late republic like the seduction or rape of an unmarried woman, entitled the father or husband to sue the man for damages and not only to divorce the wife but to retain part of her dowry. Magistrates occasionally proceeded against adulterers/adulteresses.
Augustus in the Julian law on repression of adulteries (lex Iulia de adulteriis coercendis vel sim, Dig. 48.5), passed apparently shortly after the marriage law of 18 BC, made illicit sexual intercourse (extramarital intercourse by and with a respectable free woman ) a crime, to be tried by a special court under a praetor (in practice often the senate). The law detailed restricted circumstances in which homicide by father or husband was justifiable. The normal judicial penalty for adulterers was relegatio (banishment) to different islands, and partial confiscation of property and dowry (one half). The husband with clear evidence had to divorce or be liable to a charge of procuring (lenocinium; penalties similar). On divorce, husband or father might bring an accusation within 60 days, or anyone within the next four months. A woman might not be accused while married.
Penalties were increased by Christian emperors. Constantine I introduced the death penalty (which Justinian confirmed), but allowed only the husband or the wife's relatives to prosecute. Adultery by the husband was not adulterium (unless his partner was a married woman), but his intercourse with a respectable unmarried woman (or male) constituted stuprum and in the 5th cent. AD (Cod. Iust. 5.17.8) his adultery in the matrimonial home or with a woman entitled his wife to divorce him without incurring the penalties then imposed for unjust divorce.
If you want to know more about divorce laws in Ancient Rome, just let me know!
(EDIT: a few spelling mistakes)