The Emperor Maud'bib did not, in fact, use atomics against House Harkonnen, but only deployed it against a natural geographic barrier that is the shieldwall. Had he actually targeted human beings with atomics, then he would have being in the breach of the great convention and being brought up for judgement before the Landscraad. But the laws of the Great Convention never forbid the usage of atomics against natural geographic barriers.