There have long been indications that those who go to war are concerned about the morality of it, and of its conduct. Just war theory is one specific area of study that appeals to legal theorists, philosophers, and historians, and which deals with differing views on how and when war can be conducted. Views range greatly and cross over even more disciplines (such as political science) when one considers the international relations aspects of these debates. Thucydides, for example, had a much more blunt view of how to conduct war (what we associate in political science with the "realism" school that he is essentially the father of, and which essentially focuses on power, not morality). Thucydides has been quoted and quoted again, but one of the more commonly known discussion is of the Melian Dialogue, where Thucydides chronicles Athenian generals describing to Melos that their options are to either surrender or be destroyed. From here comes the rather famous line that there is no justice between the weak and the strong, and that the strong take as much as they can and the weak have to deal with it, essentially. In short, Thucydides argues that war is simply how it is, and destruction is warranted if destruction of the other party is in the best interests of the more powerful party. In that vein, one could easily see that perpetrating "gross crimes" might even be more beneficial, because it would leave no survivors to oppose the victors in a future time.

Just war theory extended far beyond this. Consider, for example, Saint Thomas Aquinas. He sought to meld reason and faith, arguing that peace must always remain a motive of violence, and called for charity and mercy. But these were not based, usually, in some view of the benefits of mercy in the real world, but rather in the view of how it affected the spiritual. It is true that there have been historical thinkers who pointed out that if one is going to commit these kinds of acts, they had best be sure they were completed and would not lead to retribution later; this thought process was perhaps even more common than the view that these crimes should not be committed out of some moral imperative. If you're interested in further reading, I would definitely suggest investigating more on just war theory and its development; there are some fantastic books on the subject, and modern just war theory is an interesting philosophical investigation that delves into subjects with relevance to international law today.

Much of the warfare of the Early Modern Period was governed by rules, though, to quote Pirates of the Caribbean, "they're more like guidelines, really."

The savagery of the Thirty Years' War left deep scars on European memory. Large armies of crudely trained amateur soldiers had cut swathes across much of Europe, particularly the Germanies, in brutal religious wars. During these conflicts, Gustavus Adolphus of Sweden (and to a lesser extent, Maurice of Nassau) laid out what became the foundations of military practice from the late 17th century through to the wars of the French Revolution. Both these general-kings used smaller, better trained, combined-arms forces to defeat larger armies. Within a century of the end of the Thirty Years' War, Europe's various military powers had rebuilt their armies on the Swedish model: lines of soldiers armed with muskets as the main body of the army, supported by artillery, with cavalry to scout, flank, and pursue the enemy after their battle lines were broken.

In the late 17th and 18th centuries, military leaders layered additional customs and ceremonies onto this basic frame. Much of the fighting during this period occurred in the flatlands of northern France and the Low Countries. Without much by way of natural obstacles for defenders to utilize, a network of fortresses and fortified towns dominated the landscape. These fortifications, and the armies that moved between them, were supported by an extensive logistic network, since few armies lived off the land as their Thirty Years' War predecessors had. Soldiers could be, and on occasion were, executed for plundering.

Many of the rules of war stemmed from the process of besieging or defending these fortresses, which could be massive and complex. Belligerent parties needed to establish rules concerning how to communicate between besieger and besieged, at what point it was honorable for the defender to succumb to the inevitability of besieging cannons and encroaching trenches, and what would become of surrendering troops. While these agreements were worked out individually by the commanders in each case, they conformed to certain norms. Honorably defeated defenders who had held a fort until their besiegers had blown a gap in the wall could expect to march out of their fort with their arms, their flags flying, and their musicians playing a song from the victors' home country. Depending on the terms of their surrender, the defeated could march home under the promise that they could not take up arms against their victors until they went through a prisoner exchange, for a period of years, or until the war ended. In other scenarios, enlisted prisoners would report to prison camps, supervised by a small number of volunteer officers while the rest of the officers were paroled home.

All of this generosity and complexity was underwritten by a simple truth: breaking the rules or causing violence where one was needed would be met with swift, equivalent force. If the defenders of a fortress, for example, forced the attackers to assault the fort even after its walls were irreparably breached, the attackers reserved the right to kill every living thing left in the fortress. Cruelties towards POWs were answered in kind once discovered, either on a person of a equal rank or on a equivalent "worth" of lower- ranked soldiers. Beneath all the silk and lace lay hard, sharp steal.

The American Revolution saw a number of instances where the warring parties discussed or renegotiated the traditional rules of war. At first, the British felt free to abuse American prisoners: since they weren't real soldiers, dressed and uniformed like them, they should not enjoy the benefit of the military protections. As such, the British threw hundreds of American prisoners into dangerously overcrowded prison hulk in New York harbor. American victories at Trenton and Saratoga, however, suddenly meant that the British had to restrain themselves, lest they open themselves to retaliation. A particularly sharp exchange over all this was the Asgill Affair of 1782. It all started when a Loyalist, Philp White, died in Patriot custody. In response, loyalist militia captured an American officer, Joshua Huddy, and hanged him where the Americans would find them, pinning a note to his chest which read "Up Goes Huddy for Philip White." Washington immediately wrote to his British counterpart, Sir Guy Carleton, that he execute a British officer chosen at random by lot unless the British turned over the men who murdered Huddy. The luckless young man chosen for the noose was Captain Charles Asgill. The affair ran the risk of starting a chain of bloody reprisals. Asgill's bad luck ended up working well in everyone's favor. Asgill's mother was a French Huguenot refugee who had fled to England, and she still had some connections at the French court. Lady Asgill wrote to both the French and British monarchs, and Washington was able to end the whole affair by sparing Asgill as a personal favor to the French.

Washington also did not extend the courtesy of the rules of war to Native Americas. In his orders to generals Clinton and Sullivan on the eve of their expedition into Iroquois country, Washington gave these instructions:

The Expedition you are appointed to command is to be directed against the hostile tribes of the Six Nations of Indians, with their associates and adherents. The immediate objects are the total destruction and devastation of their settlements, and the capture of as many prisoners of every age and sex as possible. It will be essential to ruin their crops now in the ground and prevent their planting more.

I would recommend, that some post in the center of the Indian Country, should be occupied with all expedition, with a sufficient quantity of provisions whence parties should be detached to lay waste all the settlements around, with instructions to do it in the most effectual manner, that the country may not be merely overrun, but destroyed.

But you will not by any means listen to any overture of peace before the total ruinment of their settlements is effected. Our future security will be in their inability to injure us and in the terror with which the severity of the chastisement they receive will inspire them.

Since the Native Americans did not abide by the European rules of war, Washington apparently had no qualms about issuing orders that seem to border on genocide.

Sources:

Russ Weigly, The Age of Battle

Maya Jasonoff, Liberty's Exiles

Fitzpatrick, ed. The Writings of Washington.