Furthermore, is there any good books on the subject of ancient Satanism?
As in worshipping the devil? Not very common. First off, Satan is very weak in traditinal Judaism, especially early forms of it. HaSatan just means the adversary and he doesn't reign in Hell (the bad afterlife in the earliest texts is haSheol, simply "the Pit", and is closer to Hades than the later Jewish and Christian imagery about Gehenna). In Job, for instance, Satan is part of the Heavenly Court and basically double dares G-d to test Job.
In the Gospels you see Satan with a much more active role, tempting Jesus for example. With Gnosticism (prominent let's say in the second through forth centuries CE), you may see some actual religious relationships with what could be called the devil because Gnosticism had everything (it was many small groups with theologies that often diverged rapidly from one another). In Medieval Christian and Early Modern Christian times, you see many people very afraid of the Devil, and many people accused of making deals with the Devil or worshipping him (including, for instance, all Muslim--the name of the demon Bahomet is an obvious corruption of Mahomet/Mohammed). We see lots of accusations of Devil worship, and lots of trials, but quite frankly most of them don't make any sense. For example, the Catholic Knights Templar as a group were accused and convicted of Devil worship, despite the fact that the were one of the main groups supporting the Crusades--coincidentally at the same moment that the King of France (who initiated the persecution) was deeply in dept to the group. Many members confessed to worshipping "Bahomet", under torture. I'm not sure how seriously we should take those. Individual and occasionally devil worship, or people making deals with the Devil while still thinking themselves Christians, would not surprise me in this period. After all, these people have been told repeatedly how powerful the Devil is, how unsurprising it is for people to give in to him, and how omnipresent he is waiting for a slip, it therefore is easily imaginable that the hopeless might occasionally make a religious offering of some kind to the devil or demons as a sort of "spiritual creditor of last resort". Still, I don't think we have any convincing evidence of real, developed corporate or institutionalized congress with devil that would imply a Satanism, and certainly not long traditions of it.
Muslim and Jewish devil-worship makes less sense, because they consider evil in a different way. For instance, in the Musim world there are myriad djinns (not all of whom are evil) and there's also Iblis/Shaytan ("Satan" is a fallen angel who created the djinns in some stories; he's definitely associated with bad djinn, who can also be called shaytan). but there's not really any benefits associated with worshipping any of the evil spirits, as far as I know. They're much more tempters, the sowers of doubt, or the malicious spirits that can cause harm to befall you.
As for Atheism of any kind, there's not really any atheism in the West (including the Islamic World) until much later than you think. There are some cases in the classic world, some questionable cases in the very early Islamic World (the "atheists" of this period seem more like deists), but we really don't have records of avowed atheists until people like Matthias Knutzen, Kazimierz Łyszczyński, and Jean Meslier in the 17th and 18th centuries. And these were all isolated individuals. We don't really see atheists coming together, really, until the 19th century.