Japan did use chemical and biological weapons. The Japanese spread cholera, anthrax, and bubonic plague in China, as well as chemical weapons. They began experimenting on Chinese, Korean, and even Soviet prisoners well before Pearl Harbor (and Americans and other Allied prisoners after they were attacked by Japan), and if you're looking for some gruesome reading check out "A Plague upon Humanity: The Hidden History of Japan's Biological Warfare Program" by Daniel Barenblatt.

There's also a clinical but informative treatment of Japan's human tests here, from "The Oxford Textbook of Clinical Research Ethics."

Biological and chemical bombs were tested, as were shells, but Japan got its best results from spraying bacteria suspensions, as well as airdropped fleas and rats, over enemy cities. Japan's experiments and "field tests" killed at least 200,000 people and maybe two or three times that number. The vast majority died from the spraying campaign, but many died in the experimental facilities of Japan's Unit 731. Others were killed by boobytraps - disease-laced clothing, food, and candy.

Further, Japan deployed hydrocyanic acid grenades against the western Allies during the India-Burma campaign - basically a glass sphere holding acid which, when exposed to air, would form a deadly nerve gas. The fragility, cost, and unreliability of these weapons kept them from being widely deployed.

First of all, all the countries in ww2 had ample supplies of chemical weapons and the means to deliver them, even far into enemy territory. Any usage of chemical weapons by one warring party would have been paid back in kind. You would not get any advantages from using chemical weapons.

Secondly, poisoning a water supply is not as easy as you think. Most poisons are degradable and lose their killing potential rather quickly - they need to be used after production. Also, the amount of poison needed to effectively make the water reservoir of a major city poisonous would be absolutely massive - nothing you could just smuggle in. Several gasses and poisons are actually less potent when mixed with water, too. Most cities in ww2, especially in Eastern Europe and Russia, also did not have a single water piping system. Many used a lot of different wells - so if you wanted to poion a whole city, you would not only need to smuggle in massive amounts of poison (we're talking many tons here), you would need to distribute it quickly and make sure it kills everyone, but not on the first sip, as anyone behind the first person drinking would be suspicous of someone dropping dead from drinking the water.

Poisoning one person person is not complicated. Poisoning a city is.

Simple answer, the 1925 Geneva Protocol closed the loopholes in the earlier Hague convention's prohibition against chemical warfare. The Versailles Treaty had dealt harshly with the German use of those loopholes to introduce chemical weaponry into WWI, so all the signatories of the 1925 protocol were well aware of the penalties for violating international law.

That doesn't apply to the US or Japan though, neither of whom ratified the 1925 protocol. So, the Japanese experimented with chemical and biological warfare in a largely legal way. I believe it was because of the Japanese refusal to sign the protocol that the British gave serious thought to using mustard agents during firebombing raids, and told the US how to pull it off.

The US, for their part, simply had no desire to use the agents first. Their refusal to sign the protocol was in line with the interwar isolationist sentiment, but was also influenced by the developments in chemistry at the end of WWI. The primary persistent chemical agent that everyone used in their plans was the German Yellow Cross, or Mustard agents. The primary feedstock to produce the mustards came to be petroleum, and the US with control of the majority of the world's petrol could outproduce any nation that wanted to deploy CW. I'd describe the situation as a one-sided pre-nuclear MAD. The Japanese knew this, and from what I recall reading they purposely removed their chemical warheads from any theater in which the Americans operated to avoid an unpleasant escalation.