A lot came from Romans themselves actually.

Cato the Elder was noted for his campaigns against Moral Degeneracy (Though he said it was Greeks that were responsible for corrupting Romans), and it was a common practice as well for historians under pay of an Emperor to slander another Emperor or another dynasty as being corrupt or immoral, so as to look better by comparison.

The word Censure came from a Roman position as well. The censor was responsible more or less for "Maintaining Roman Values", and they led various campaigns against sex, corruption, etc.

Then, of course, the Church used the idea of orgies and decadence as a way of portraying the Pagans and others as sinful, and to spread their influence and gain converts that way

The original trend began with Rome's annexation of Greek territories. What up until then was a somewhat culturally conservative society with strong rural heritage was exposed to the flood of cultural influence from the conquered Greeks: art, literature, educated Greek slaves, food came pouring into the city. It was readily adopted by many Romans, and anything Greek quickly became seen as fashionable, refined, sophisticated and cosmopolitan. Some more conservative members of the Roman polity, however, understandably were put off by such drastic changes to the city's way of life. To them, Greeks (and thus, anyone who tried to imitate them) appeared weak, effeminate, a danger to the old traditional values of the state. So, already as early as the middle of Republican period of Rome you can find the accusations of deviancy, and condemnations of effeminate behaviors.

During Augustus' reign he implemented a program to revive the old Roman "virtus" (virtue), taking steps to correct what he saw as tendency towards deviancy and weakness. Exile of such major figure as Ovid, is one notable consequence of this policy.

Later, of course, the ascendant Christians take on this role, and it is their criticism that is most prominently seen today. But like we've seen, their call for moral purity was not unique.

As to why popular culture today associates "kinky sex" with Rome, that is likely the consequence of Christian Church's writing. Ultimately, they became the dominant power in Rome, and this were at leisure to dictate the overall narrative. It was in their interest to position themselves as beacon of light and salvation amidst the crumbling city of filth and degeneracy. From such position they could comfortably write a lasting record of events from their perspective.

One thing that I haven't seen mentioned yet is that it was quite common for ancient Romans to use examples of sexual debauchery as an example of a public figure's more general moral debauchery. So pretty much anytime you see an account of a really unpopular emperor (or at least unpopular with the surviving sources) you'll inevitably see a bunch of lurid stories about him doing nasty things to people. Those stories might be just political exaggerations or wholesale fabrications, but nothing sticks into the public consciousness like filthy stories.