I've learnt about a lot of different battles from many regions and many times and massacring most of or maybe all of the entire military and civilian population of a city or region is one of the most common trends I've seen and this trend goes right back to the beginning of the Bronze Age and human warfare.
It's easy to say things like humans were vindictive or barbaric or lacking in morals or driven by bloodlust or something else presumptuous along those lines. But I think such explanations are just shallow and they wouldn't cause such a widespread trend. Surely there is a more strategic and necessary reason to abandon your compassion and kill masses of innocent noncombatants, men, women, and children.
I was thinking about this because I was playing a particular game called Unciv. I guess it's supposed to be an android copy of the Civilisation game series where you have to create an empire essentially from scratch whilst competing for dominance against other empires. During a war I was having, I had the options to puppeteer, annex, or raze one of the cities. At first I thought why would I just burn this useful city? So I annexed it. And then the city became utterly useless. It was worse than useless because occupying the city was draining my resources whilst not providing any in return since the population of the annexed city were in open defiance and refusing to work and the situation continued for some time and took its toll on my economy. It cost me more to annex it than it did to raze it and that's when I realised why the option for razing was there.
I thought it's probably more likely that killing innocents was only necessitated by protecting your own innocents as it prevents rebellion, it sends a message, but also, more importantly, it damages their economy and it prevents the defeated enemy from seeking revenge because, with a smaller population, it's harder to raise an army and to recover economically. And these advantages may have just naturally led to the brutal trend of civilians being targeted and killed. Is there more on the topic about why exactly the trend was so common?
Let's take a look at Genghis and Caesar, while keeping in mind records this ancient are impossible to verify. Still, I think it should provide a sufficient answer.
After his conquest of the Western Xin dynasty, he began engaging diplomatically with the Khwarazmian Empire. Long story short, he ended up invading.
At the city of Bukhara, some records indicate the burning and massacre of the city was in part due to a holdout of resistance in the city's primary citadel, and so the result of collateral damage. It's recorded that Genghis even gave a speech in the square which became religious justification for his massacre, that he was sent by God to punish them, though that was almost certainly not his reason.
So, at Bukhara, if his reasons were not to weaken local resistance, then it was collateral damage from a type of ruthless warfare that didn't take into account the moral conundrum of innocent deaths. This is not the best example of complete extermination, as it's indicated that Bukhara quickly recovered and even benefitted greatly from their Mongolian hegemons. It is, however, an example of how death on such a massive scale was possible in the absence of more modern ethical concerns.
Other cities did not fare as well. He and his sons destroyed and slaughtered the citizens of several other cities, but only those who fiercely resisted, with most estimates claiming millions dead. No city with no people means no revolt.
Similarly, in Gaul, Julius Caesar had spent a decade building relationships with Gallic allies and conquering the resistant tribes, until many of the banned together under Vercingetorix to resist Roman rule. Here, when Caesar went to put down these belligerents who had begun raiding Roman allies and being generally disruptive, he was particularly unmerciful. After treating with several tribes, he immediately attacked by surprise, which saw the extermination of at least two tribes.
The rest of the Gallic wars were marked with similar instances.
Here, for the same reason as Genghis and his sons, we see genocide for the sake of subjugation. These types of slaughters discourage resistance, and the sparing of willing subjucts encouraged compliance.
A city or a people are more trouble than their economic boon is worth to the hegemon if they will revolt in a decade.