Modern revisionists such as Winterling firmly reject the innuendo of Suetonius and Dio. All of the contemporary sources of Caligula and in the immediate aftermath such as Seneca make no mention of it.

I personally believe that a deeply misogynistic and gossipy Roman aristocrats who wrote the histories looked at stories of Caligula's relationship with Drusilla and interpreted sexual gratification as the only explanation for such a close relationship.

When Caligula was at points unmarried Drusilla sat on his couch with him, the position typically reserved for his wife. He likely simply favored her the most of his sisters and was devastated by her death to the point he proposed she be worshipped as a goddess.

Similar unfounded slander would be put on Domitian for his close relationship with his niece, Julia.