Well, most classic sources are pretty harsh regarding the demeanour of Gaius Caesar Augustus Germanicus. Keep in in mind that his rather short rule of four years saw numerous senators sentenced to death for high treason it is hard to argue that at least something went rather awry. On the other hand there are modern interpretations that are more sceptic about the sources for the reason you also imply: after he was murdered by his own guard there definitively was a campaign to make him a really nasty villain.

A translation of Philon of Alexandria can be found here: http://www.earlychristianwritings.com/yonge/book40.html and it fits the smear writings in many things though it is hard to believe that there is not also a kernel of truth about the man called Caligula. There are also works by Flavius Josephus, Sueton and Cassius Dio, but those wrote later after the death of the emperor.

I think the following article sums it up pretty nicely: https://www.bbc.com/news/magazine-23455774 - some of what was written about Caligula may well be hyperbole or propaganda but it is hard to believe that some of the anecdotes are pure fiction.