I would say this is the type of question for which no definitive evidence exists to conclusively prove or disprove impropriety... a quick look at what evidence exists, though, about the relationship between these men would indicate that it was probably not sexually abusive.

Quick background, Gaius Octavius, later Octavian and later still Augustus (as the first emperor of Rome after 27 B.C.) was the great-nephew and adopted heir of Julius Caesar. His mother, Atia, was JC's niece, the daughter of JC's sister.

I have not read Gaiman's book, but the chief ancient source of these allegations was Mark Anthony as told by Suetonius. MA claimed that GO was the main beneficiary of his great-uncle Julius Caesar's considerable estate because of sexual favors. Suetonius, writing over a century after GO's death, after mentioning the story, claimed that this was political slander, one of several false stories about licentious behavior on the part of GO meant to discredit him. Again though, this is heresy, with no means of verification. Mark Antony had served under JC as a military commander, and he and GO had a history of acrimony after JC's death that included, among other things, MA throwing over GO's sister Octavia the Younger in favor of Cleopatra, and culminated in full-on war (a war won by GO). While there is no proof (outside of Suetonius) that MA even made these allegations, the history between the two men suggests it is quite possible he did so while seeking to tarnish his chief rival's reputation...

Julius Caesar had no heir, and it is likely he adopted his great-nephew as heir for the very simple reason that he was a proven up-and-comer (like Caesar himself). GO was largely raised by Julius Caesar's sister, Julia Caesaris, and when she died he delivered her funeral oration (52 or 51 B.C.). In 46 B.C. GO was put in charge of a series of games hosted at a temple built by JC. Clearly the young GO was a visible figure within the family. He requested to join JC on campaign against Pompey in Hispania, and after overcoming an illness he went. En route he was shipwrecked, and made his way through rough territory with a handful of surviving companions. Again, this likely impressed his great uncle, who was a military man and almost certainly admired strength/courage. By the time JC and GO returned to Rome, JC liked the young GO enough to change his will to name him prime beneficiary.

So, could there have been sexual relations or abuse between JC and GO? Sure, but no credible source attests to it of which I am aware, and it seems more likely that JC, having no legitimate living children under the law, simply admired the young man's drive, ability and spunk, knew him to have been beloved by his sister, and/or saw him as the only credible heir to his fortune.... most probably it was a combination of all of these things.

Disclaimer: I'm a biologist, not a historian... so any additions to this post are most welcome!

Sources:

Suetonius, Lives of the 12 Caesars (this is a great source of info on the first 12 emperors, although some of it may be conjecture, but it is also a damned fun read, full text in the link I provided in the body of my response above). Get the Robert Graves translation if you buy it.

Michael Grant The Roman Emperors

Tom Holland Rubicon