If I'm not mistaken, the reasoning behind the internationally agreed upon accords on chemical weapons were its indiscriminacy and the particularly cruel and painful deaths and wounds they caused. Perfectly reasonable. However why was this logic not applied to flamethrowers? They too made their debut in the First World War, and were used widely on the western front (also the eastern? forgive me I'm not very familiar with those campaigns) for tactical reasons.
I find it very difficult to imagine a more painful, indiscriminate, inhumane death than immolation, but as far as I know there were no such international bans in the Interwar period, and they continued to see extensive use in the Second World War and even Vietnam. I know white phosphorous does have some restrictions today but there are still several nations that have confirmed supplies of them. (I don't know when white phosphorous rounds and bombs were "invented" and when these agreements where made, so this may fall under the 20 year rule)
Why are chemical weapons pretty universally reviled, but incendiary weapons remained widely in use?
I'm not sure "indiscriminate" means what you think it does in this context. Chemical (and biological) weapons are considered indiscriminate because it is very hard to use them in a way that discriminates between legitimate and illegitimate targets, soldiers and civilians. Chemical weapons are weapons that target a wide area, which is a bad start. Worse than that, they can easily be blown off their intended target. That can be seen in the examples of gas being blown onto friendly lines. In other circumstances those chemical weapons could just as easily have blown into a town.
In contrast, flamethrowers have a relatively small area of effect (at least, as long as they don't start a bigger fire), and are used at close range, allowing users to discriminate between targets.
White phosphorous rounds are today politically sensitive, so domestic policies such as the British Army observes may limit them, but they have no legal restrictions on use, bar two. Unfortunately, the fine details are not usually observed by commentators and are subject to much urban myth, even within the military.
To keep it simple, the US does not issue ammunition which may be legal in some situations and not in others. It doesn't want troops worrying about that sort of thing in the heat of the moment: If it's issued, you can use it. WP, which was a common round in WW2, is categorised as an incendiary munition. As such, the only 'unusual' prohibition (i.e. not subject to the normal rules like "Don't use it on civilians", "don't use it on prisoners etc") is the prohibition on its use as an air-delivered munition (i.e. dropped by airplanes, drones, blimps, zeppelins or helicopters) on "any military objective located within a concentration of civilians." (See Protocol III of the 1980 Convention on Certain Conventional Weapons").
The other prohibition is on using its choking effect, such as it is. Against the respiratory system, WP fumes are characterised as an 'irritant', and like any such thing, you can't use it with the primary purpose of trying to choke someone. However, in terms of 'irritant', it is well below the level of tear gas, nobody has ever died from it, and it is a pretty terrible munition to use in that role, so the matter never comes up as a practical issue: If they're close enough to cough on the smoke, they're close enough to be on fire. (FWIW, tear gas is a chemical weapon under the laws, so soldiers using it in, say, crowd control in Iraq need permission from D.C.)
Otherwise, tiredstar's answer seems to be pretty much on the money. You can aim a flamethrower. You can even aim an artillery barrage or a bomb with a reasonable expectation of where it will have effects. Aiming gas, however, is impossible. Yes, burning is a pretty horrible way to go, but I'm not sure I can think of many pleasant ones on the battlefield.