Great question. This is something that has interested military historians in the last several decades, especially as the period moves away from battles and tactics towards the cultural and societal effects and logics behind war. Historicizing the way we think about war - such as what makes some part of war "normal" while others are "savage" - is a very important part of the field.

There are several books I can think of that might interest you and which will surely lead you to more sources. You said you weren't looking at any one period, but this is something Indian historians have tackled especially. One book is Frederic Gleach's Powhatan's World and Colonial Virginia: A Conflict of Cultures. Gleach argues that early colonial Virginians and their Algonquian neighbors the Powhatan confederacy fundamentally misunderstood the "language of conflict" which the other culture used. His core example is the Great Massacre of 1622, when Powhatan warriors raided Jamestown, killing hundreds of English settlers. Gleach rejects the titles of massacre or uprising for this event, arguing that they do not mesh with an Algonquian world-view, nor was the English concept of war, with all its finality, understood as such by the Algonquians. Instead we must understand the raid as a one-time “coup” intended to punish European transgressions, based on the Algonquian conception of warfare’s “corrective, moral nature”. Rather than massacres, says Gleach, the events were the “severe parallels to the slap on the wrist." Other historians have accused Gleach justifiably of splitting hairs - killing hundreds of people is certainly a massacre - but still he is right that whereas the English understood the attack as a declaration of total ethnic war, the Algonquians saw it as a one-time rebuttal to English provocations in region. That is the way another people in the area would have read it, but not the English, who believed they had become involved in a kill-or-be-killed war.

That is of course an historical case study, but one I think discusses your question. How such logics of war develop is a question Wayne Lee researches in his book Barbarians and Brothers: Anglo-American Warfare, 1500-1800. Lee explores how American notions of war developed: what explains a military's restraints in some areas and willingness to commit atrocities in others? Like Gleach, Lee discusses the existence of an internal logic of war. Lee argues that whereas certain institutional restraints developed in European militaries which fought each other during the 16th century, they did not against enemies who were perceived as "savages." That is to say, during the 16th through 18th centuries, European armies grew better-organized and disciplined, while Enlightenment thinkers sought to codify and govern methods of war to constrain violence. If prisoners could be expected to be treated fairly, for example, then soldiers need fear the consequences of defeat less and surrender more willingly, reducing bloodshed.

On the other hand, British soldiers fighting in Ireland, and then Anglo-American troops in the colonies never extended this restrained "logic of war" to their enemies, whom they perceived as savages. They denied the possibility their enemies could fight in a "civilized" manner and therefore waged war on them in totality. When Irish or Indian enemies resisted, it merely confirmed Anglo-American ideas of superiority.

Thus Lee can explain how very similar group of American soldiers fought a restrained campaign against the British during their War of Independence, while conducting a war of scalp-hunting extermination against local Indian enemies. Another example Lee gives is Confederate troops in the American Civil War, who fought their Yankee foes "without hate," unless they were black, and in that case showed them no quarter. Or one final example - note how the majority of Indian victories against white troops in American history are labelled as "massacres," while Anglo-American victories are always battles. This is not because the battles were actually conducted differently, but because of historical narratives have stressed the triumphal march of white civilization versus Indian "savagery."

The important thing to take from this is something you are clearly already well aware of: logics of war are not natural; they are historically produced, in this case from a history of racial prejudice and settler colonialism in Ireland and the Americas which portrayed Anglo-American violence as "civilized" and that of their indigenous enemies as "savage," thus justifying and equally savage response.

I have discussed historians who look at early American and Indian history, but of course these are still issues which are much debated today. Up through the Second World War it was considered necessary that prisoners of enemy nations be treated well (with the Japanese being another exception, and for a fantastic consideration of this case I cannot recommend enough John Dower's War Without Mercy: Race & Power in the Pacific War, which will also discuss this topic at length.) But enemy fighters the West takes prisoner during the war on terror have upended normal understandings of war because they belong to no state, and that has been used to justify torture and inhumane treatment upon them.