How do we know the "crazy" stories about Roman Emperors aren't just old propaganda?

"Propaganda" is probably not the correct word, as it is difficult to say that, say, Suetonius was trying to influence mass opinion towards a particular position when he wrote that Tiberius liked to bathe with naked boys. But I get what you are saying, and the answer is, broadly speaking, we don't. To take a pretty prominent example, Nero is widely derided in the sources for being an effeminate, homicidal Hellenizing girly man, and you get the sense that nobody likes him. But after his death, there were a number of provincial rebels claiming to be him and seeking to restore his position--why would they do this if everybody hated him? (of course you could argue that the particular emperor didn't really directly effect conditions in Achaia, and so the pretenders were just articulating local grievances in an idiosyncratic way, or perhaps displaying support for the Julio-Claudian line generally rather than Nero specifically--this can go around and around). Anyway, the dispute about Nero's memory is something that exists today, and there are a number of scholars who argue in his favor.

Caligula is a bit more complicated, so there isn't really much of an active apologetic cottage industry for him as there is for Nero but there are a few dissenting voices(the early twentieth century classicist John Balsdon being an example). We know that there were some ancients who wrote favorably of him whose works are lost, and it isn't super difficult to construct alternate explanations for his more lurid deeds--for example, perhaps the (proposed) elevation of the horse Incitatus was not a sign of insanity, but rather a public rebuke of the Senate, a way of saying a horse could do this job.

I can go into a few more examples (for example, Pliny did not really care for Domitian, even though Domitian did nothing wrong) but I think in general "propaganda" is the wrong way to think of it, rather that our sources have particular perspectives on matters.