Whilst I'm probably not qualified to give a hugely detailed answer (I'm not a historian - merely an International Relations graduate), I can hopefully shed some light on this.

The "rules of war" are quite slippery business - and as I'm sure you're aware shift over time. As the international system is 'anarchic' - in that there is no entity with power over all states and certainly no means to enforce its will over all - the rules have instead been built up over time out of precedent and agreement.

What enforces the rules though is essentially down to a game-theory style calculation on the part of all of the participants. For example, if one of the sides in WWII had decided to indiscriminately start killing prisoners of war, the other side would have done so too - so pre-empting this, neither side wants to break the taboo, lest they also suffer the consequences. Similarly with regard to using nuclear weapons during the Cold War - no one wanted to be the first to use them, because then the other side wold retaliate in kind (but conversely, both sides had to act like they were more than willing to use nukes first, in order for the expectation to remain).

So in a sense, rules can be relatively self-enforcing.

There's also the question of legitimacy - as we've got documents like the Geneva Convention and the Nuclear Non-Proliferation Treaty, these documents can be used to undermine the legitimacy of the state's actions to outside players. Which could affect how the war proceeds.

For example, if one side is killing civilians en masse and the other isn't, then the opinions of all of the other states not involved may shift against the state that is deemed to be breaking the rules.

When Iraq invaded Kuwait in 1990, it broke one of the biggest 'rules' in the international system - don't intervene in another nation's affairs. This led to the first Gulf War in which a huge coalition of other countries not only got involved to defend Kuwait, but also to defend the norm of non-intervention.

So it is also important for a state to play by the rules, so it will be viewed as a legitimate participant in the international system - if it breaks too many of the agreed norms and precedents, it could be viewed negatively which could lead to poor consequences economically, diplomatically, etc, from countries that are not (directly) involved.

The dangers of using chemical and nuclear weapons or killing POWs were that they would also be used against your own people. No one wants massive civilian population dying, if this occurred there would probably be destabilizing of the governments due to non approval from the people. In other words those in control of the nations power would risk the security of their position if all rules went out the window and people were killed indiscriminately. The others answers are more thorough but it is also important to see that it would be unwise for leaders of their respective nations to break these "rules"