I don't know anything specific about mustard gas development. However in the United States, a massive research establishment for poison gas research was created on the campus of American University (in Washington, DC) starting in 1917. It involved some 60 campus buildings and employed some 1,700 chemists. They used a lot of animal testing to determine the toxicity of various chemical compounds and countermeasures to them. The US then decided to get into the production of Lewsite, an arsenic compound several times deadlier than mustard gas. To produce it, they first developed the laboratory process at AU, then exported it to a commandeered automobile factory near Cleveland, Ohio, where it was carefully produced in quantity, under conditions of secrecy so great that it was known as the "Mouse-Trap" by those who worked there (because nothing that went in could come out). The deadly product was not completed in time for use during the war, and was sunk off of the coast of Baltimore. The production was overseen by the Harvard chemist James Conant, who later went on to have an important role in the development of the atomic bomb (and the opposition of the hydrogen bomb). I have written about this at length here.

I provide this just as one national model for how this kind of research was done. The Germans started with gas work under the direction of Fritz Haber, one of the most important chemists in history (for the Haber-Bosch process, which has nothing to do with gas weapons), working out the Kaiser Wilhelm Institute for Physical Chemistry and Elektrochemistry in Berlin, in cooperation with the world-renown German chemical industry. I would imagine they did something similar — massive laboratory work, animal studies, and finally field studies. But I don't know the details of the German R&D.