Specifically, who/what sets the rules of engagement? For example, I recently read that during the beginning of WWII submarines would surface before they engaged other ships. Why is this? Are there actual laws that set this precedent or is it just an unspoken agreement between nations at that time?
The legal requirements come from two major sources. International treaty, which is the minimum possible, and national law.The major treaties in force are the Geneva Conventions, The Hague Declarations, and a couple of additional multilateral treaties such as the Ottawa Convention, the UN Conventions and so on. Those are the obligations of one party to another and can be considered war crimes if broken. National laws apply only to that nation's military.
The next level down is the rules of engagement. These are set usually for political reasons, though sometimes operational ones, and although breaking them can be a punishable offense for the offender, they are not war crimes and the enemy forces have no reasonable expectation that they will be followed.
A topical example, for example, is the use of white phosphorus artillery rounds. There is no blanket legal prohibition on using the munition against personnel or equipment. However, politicians don't like the media showing video of hideously burned corpses. (Witness the Highway of Death of the 1991 war). As a result, it has become regulation in a number of militaries (I believe the UK is one such) to restrict the use of the munition to screening (smoke) purposes. As a caveat, there is a prohibition in the Geneva Convention to the use of the munition in instances likely to cause civilian casualties.
So, for the hypothetical, an enemy facing the British Army (if my recollection is correct) may very well expect to not have to face WP used in the anti personnel role. If he's hiding in an area likely to cause disproportionate civilian casualties and the British use it, it's a war crime. If he's hiding in an abandoned town or forest or some such, and the British use it, the soldiers may be charged with disobedience of orders by his own chain of command, but the opposition has no legal recourse as under applicable international law, there was no prohibition.