Caligula's full name was Gaius Julius Caesar Germanicus, son of Gaius Julius Caesar Germanicus, whose name before his adoption by Tiberius was Nero Claudius Drusus Germanicus. After the death of Tiberius Caligula was granted the title Augustus by the senate. Those are all perfectly acceptable versions of his name.

I don't know what translation you're using for Josephus and Suetonius, but neither of them call him Caius. Suetonius calls him Gaius, except when he abbreviates his name to C. Caesar. "C." was the standard abbreviation for the praenomen "Gaius," a holdover from the time when, in Archaic Latin, the letter "C" was used for both g and k, since those are simply the voiced and unvoiced forms of the same sound. There's nothing odd about abbreviating Gaius to C., and it's not to be read as though it were pronounced "Kaius." Sometimes in texts you'll see Gaius written as Caius for Romans of older periods, but that's deliberately archaizing--the old Loeb, for example, of Plutarch's Marius wrote his name out as Caius Marius for some reason, even though Plutarch writes Gaius (Γαΐου Μαρίου τρίτον οὐκ ἔχομεν εἰπεῖν ὄνομα). Josephus also writes Gaius (Γάιος δὲ οὐκ εἰς μόνους Ἰουδαίους τοὺς ἐν Ἱεροσολύμοις...), at least in my edition, unless some of the minor manuscript traditions have something different...