The answer is yes and no.

Many of the tales that appear in culture today, the excesses of people like Nero and Caligula, aren't really representative of Roman society in the same way that the lifestyles of the Kardashians and the Hiltons are representative of ours. A lot of the myths of Roman sexuality were passed down to us by early Christians, so that bias needs to be taken into account.

The Romans had a different view on morality than their Christian successors. For example, divorce was very common, especially in patrician (noble) families and prostitution was both legal and prosperous. Romans regularly used symbols of male virility (like the phallus) in art and architecture for good luck. Like the Greeks before them, the naked male form was seen as the epitome of beauty, and was depicted in sculpture and art.

Roman society was, however, regulated by a strict morality (pudor = modesty), especially after the time of Augustus Caesar (http://www.ancient.eu/article/116/) who encouraged Romans to stay monogamous and have large families, by making stricter laws against adultery and rewarding families with more than three children.

The more complicated answer is that Roman's had a different view of sexuality than the societies that came after them, and also that view could change drastically depending which part of the Empire you were in, and at what time.

The Wikipedia article on this is actually quite good, with lots of great references at the end: https://en.wikipedia.org/wiki/Sexuality_in_ancient_Rome