We spoke about him today in class and I became pretty interested in him. I wouldn't be surprised if the whole "waging war on the ocean" and "naming his horse senator" probably aren't very accurate, but I was curious to know more about his rule. Specifically the kinds of mental things he might have done to spark such stories about him.
The 12 Caesars by Seutonius. Roman history by Cassius Dio.
Seutonius is very much into scandal and gossip, but that has the advantage of discussing personal matters that more serious scholars might ignore.
Also The Annals by Tacitus, who is a better historian. This gives a good background about the general period but the sections on Caligula have been lost.
I always prefer in reading the original sources and these books are well translated and entertaining.