There was certainly a lot of uncertainty, but the uncertainty was bounded. Here is a very abridged rundown of what I think the interesting areas of uncertainty were:
To get it out of the way, very minor uncertainty about whether the test might ignite the atmosphere in a giant fusion reaction. Which is to say, they were almost totally sure that wouldn't happen. Almost is not 100% certain, but they did not really think it was a realistic possibility. But it isn't as uncertain as it sometimes is portrayed.
They had considerable uncertainty about the efficiency of the implosion mechanism being tested, which meant they had considerable uncertainty about the explosive power of the bomb. The "solid" bet was around 4-5 kilotons of power. The pessimists thought it might not work at all. The "optimists" thought it might be several tens of kilotons. In the end, it was 20 kilotons, which is 4-5X more powerful than they expected. That's not terrible (especially since damage is mostly an issue of order of magnitude; you have to increase the explosive power by a factor of 8 to increase the damage by a factor of 2), but it's still an uncomfortably large factor to be off by when we're talking about a nuclear weapon.
They had a lot of uncertainty about where the radioactive cloud would go, or how intense it would be. They had monitors and soldiers in nearby towns, ready to evacuate them if the radiation levels got unacceptably high. They didn't end up evacuating anybody but a considerable area was exposed to fallout.
They had a lot of uncertainty about how nearby people would react, whether they would buy their "an ammunition dump" exploded story, and so on. Would their secrecy hold? They hoped so, but there wasn't much they could do about it beyond what they did.
They had a fair amount of uncertainty about the actual effects of the weapon. They could guess how some of the effects (blast, radiation, pressure) would scale up to kiloton ranges, but they really didn't know. They tried to use instrumentation to gauge it; some of it succeeded in getting good data, while some of it failed (a bunch of it was destroyed by the test itself). This means they went into the bombing of Japan with only a rough idea of what was going to happen to the cities they were dropped on (destroyed, yes, but there are a lot of other details — they didn't expect so many people would be exposed to acute radiation and survive the initial attack, for example).
There are other areas one could probe for uncertainty (if you or anyone else have questions, feel free to ask them), but this I think gives a gist of what they were thinking. I think it's important to note that uncertainty exists on a scale, which is to say, it's not all or nothing. In some areas you might have a little uncertainty, in some areas you might have a lot, in some areas you might have none, etc. Separately there are the areas where you don't even know you have uncertainty (the oft-mocked but not-dumb idea of "unknown unknowns"), which can be their own sources of deep uncertainty (an example from Trinity might be, "were there any spies at the Trinity test?", which nobody appears to have even considered at the time; at least one spy, Klaus Fuchs, was present).
The best overall book on the Trinity test is F.M. Szasz, The Day the Sun Rose Twice, which has a lot of detail in it.
More can be written, but you might like to start with "Was there really any fear during the development of the atomic bomb that it could detonate the atmosphere?" by /u/restricteddata.
If you're also interested in post-war calculations, a later discussion by the same is "I just read that, when they tested the atomic bomb, there was a belief it might light the atmosphere on fire. If they believed this, why did they still test it?". I hope that I can also point to his recent blog post -- not in AskHistorians -- "Cleansing thermonuclear fire".
This is not to discourage discussion. More questions, data and debate are welcome.