Real-enough that they calculated it was very unlikely, and some of them thought about it during the test. Not real-enough that they seriously thought about calling off the test.

The fear was that the heat of the bomb would induce nitrogen-nitrogen fusion. They didn't know exactly how much temperature it would take to fuse nitrogen. But they were able to say, look, if we guess that nitrogen is MUCH easier to fuse than we think it ought to be then we still find that this isn't going to happen. Even if it did happen in a tiny part, it wouldn't maintain itself as a reaction — it would rapidly cool. It turns out it's super hard to ignite significant fusion reactions even under ideal conditions — it takes very careful engineering to get much easier reactions (like deuterium-tritium) to burn and propagate.

Still, there were some lingering uncertainties. Nobody had set off a nuclear weapon before. They could say, well, there have been presumably high-energy activities in Earth's past (asteroids, meteors). Those aren't quite the same thing as a nuclear weapon (where all that energy is concentrated into a very small volume and is expressed primarily as radiation, heat, and blast), but close enough to make one think that if you could ignite the atmosphere, it might have happened at some point in the past.

Arthur Compton (a Nobel-Prize winning physicist who ran a major phase of the bomb project at the University of Chicago) is said to have argued that "it were proved that the chances were more than approximately three in one million that the earth would be vaporized by the atomic explosion, he would not proceed with the project. Calculations proved the figures slightly less — and the project continued." Which is an interesting bounds on the matter.

Still. Some uncertainties. My favorite expression of this comes from James Conant. Conant was not an idiot — he was a prominent chemist and the president of Harvard University. He had been involved with the Manhattan Project from the beginning. He didn't think the bomb was going to ignite the atmosphere.

And yet — when he saw the blinding flash of the first nuclear detonation, and felt the heat on his face, his brain went to scared places:

Then came a burst of white light that seemed to fill the sky and seemed to last for seconds. I had expected a relatively quick and bright flash. The enormity of the light and its length quite stunned me. My instantaneous reaction was that something had gone wrong and that the thermal nuclear [sic] transformation of the atmosphere, once discussed as a possibility and jokingly referred to a few minutes earlier, had actually occurred.

After the war, they put the equations for "how we knew it wouldn't ignite" into writing (Konopinski, Marvin, and Teller, document LA-609, 1946). Again, it's an argument about likelihood — not certainty.