Honestly, a lot of it really comes down to justifying enslavement and empire rather than any real aspects of the culture.

Philhellenism was kind of a swinging pendulum in the Roman world and Romans liked to style themselves as kind of simple country folk, not fancy talkers who like to idle. You do get the opposite extreme with emperors like Hadrian though.

That said, as far as what actual Romans accused Greeks of there's a lot of Roman suspicion about Greek sophistry being morally corrupting and overly superfluous and that Greeks were incapable of self-governance.

And on the specific issue of effeminacy, here's a very interesting linguistic note: Ancient Greek and Latin don't really have terms for hetero- or homosexual, instead sexuality is more defined by the kind of sexual acts performed and whether the person had an -um- penetrated or penetrator role. And there are all kind of complexities in Roman sexual mores that don't really translate well across two thousand years that make this much more complicated. Without going on a long excursus though, you're more likely to see a moechus (adulterer) described as effeminate in a Latin text than (what we would call) a homosexual. Because Roman terms for effeminacy mean something more along the lines of spending a lot of time around women and thereby adopting their worldviews.

Basically, a lot of modern / casual readers of history think the Romans were calling the Greeks gay, but that isn't really what's happening.