So the key questions were: 1. Is this contraption going to work at all? and 2. If so, how well will it work?

The bomb design being tested was the implosion design, a very complex piece of machinery. The working, in brief, is that an high-voltage electrical signal would be sent to 32 detonators, which would set off 32 little explosions that would themselves start explosions in a mixture of high-explosives, which would then have their blast waves shaped so the time they converged on the solid metal core in the center of it all, they would be squeezing it it from all sides, in three-dimensions. This dense, metal core would then be compressed to some fraction of its original size, increasing its density and thus creating the conditions for an explosive nuclear chain reaction.

So the issues were: 1. Will all of that electrical and explosive stuff actually work as planned when tested? There was very little room for error. The simultaneity of the whole thing had to be on the scale of tens of nanoseconds — just levels of precision that were then, and still are now, highly unusual. They had to invent the electrical system and detonators from scratch. It is easy to imagine one detonator misfiring, for example; to get around this, they had extremely high quality-assurance standards and tested hundreds of detonators before firing. 2. Will the blast wave from the explosive lenses actually meet at the core at the same time? This proved very hard for them to test at the time without a full nuclear explosion. Today we have very sophisticated ways to basically create 3D-X-ray films of an imploding weapon without a nuclear core in it; they didn't have that then. They tried, very hard, to get information that would tell them if it was working, but they ultimately, even on the eve of the test, were lacking conclusive evidence. Note that if the blast wave was not extremely symmetrical, it would just serve to "squirt" the core out the side that was not there at the right time; the metal did not "want" to be compressed into a smaller, denser ball!

Now assuming all of those things did work, how much compression of the core would you really get, and how long will the core stay together after being compressed? These are what are going to determine how much explosive output you ultimately get, and again, were maddeningly difficult to figure out without a test.

The "safe" guess as to how explosive it would be before the test was 4,000-5,000 tons of TNT (4-5kt) equivalent, if it worked at all. The scientists actually took up a betting pool on the issue; we know that Oppenheimer (0.3 kt), Teller (45 kt), Kistiakowsky (1.4 kt), Bethe (8 kt), and Ramsey (zero) put in. This doesn't necessarily reflect their pessimism or optimism (since they were apparently restricted to unique values), but it does give one an interesting spread when the scientific head of the whole project (Oppenheimer) was betting so low (300 tons of TNT is not much). It is also interesting that the head of the explosive lens development (Kistiakowsky, aka, the man who had stayed up all night removing air bubbles from the explosive lenses with a dentist's drill) didn't bet a whole lot higher than him, and that one of the people in charge of actually assembling these weapons and making them operationally functional (Ramsey) was willing to bet that it wouldn't work at all. It is easy to read into this the idea that the people who were closest to the bomb as a piece of clunky, hand-made technology (a "Gadget") were the most pessimistic about everything coming together, whereas the theorists (Teller and Bethe) were more optimistic.

Ironically, one of the last people to the betting pool was I.I. Rabi, another theorist, and he picked 19 kt out of essentially lack of choice (it was a value not taken), and he was considered the winner in the end.

Anyway, as stated, the committee that was coming up with the final yield guess (the Cowpuncher Committee) concluded just before the test that it would likely be 4-5 kt if it worked. In the end, it was more like 20 kt (the exact yield estimates range from 18-22 kt or so, and it doesn't really matter since those are all the same basic magnitude of explosion and damage). So it worked significantly better (4-5X) than expected, to the surprise of all. This had big historical implications, as an aside, because they had previously thought that their untested bomb, the uranium-235 gun type design (Little Boy), would be their "big bomb" (10-15 kt was their pre-Hiroshima guess, and it was 15 kt), and that this would be followed by a series of "smaller" bombs (the Fat Man bombs) which would be 4-5 kt, but there would be many more of them (their production line was such that they could make 1 Little Boy bomb every 2 months, but 3 Fat Man bombs per month). Finding out that even the first implosion bomb was actually more powerful than the gun-type bomb was a welcome surprise because it meant that the US nuclear arsenal would have quite a lot of powerful weapons in it from the beginning (they always assumed that with further testing and development, implosion would be the best long-term option, but they didn't think the first iteration would necessarily be so).

On the question of "did they think it would destroy the whole planet" — they did not have 100% knowledge of exactly what would happen, but they were able to do enough calculations to see that there was practically no chance of it. They could look at the possible reactions that might spread and see what their initial conditions might be and the probability of their spread, and then say, "OK, here's what we think the conditions/values might be, let's assume it was several orders of magnitude easier to start these reactions than we think, and see if it's still possible." The answer was no. They also could reason that if very large explosions could set off these reactions in the atmosphere, the world would have already been destroyed a long time ago from meteor strikes, which release amounts of energy that can make nuclear bombs look very small.

You can see from the above that their main concern was not a bomb so big that it would destroy the planet, but a bomb that didn't work at all, or didn't work that well. As an aside, they did understand the effects of radiation from the test pretty well, though their standards for exposure and risk were higher than we have today. But they did design the test with radiation safety in mind, at least the safety standards of the time.

There is more that could be said about the above, feel free to ask... some things I have written: this discusses the betting pool a bit more, a speculative post I wrote about what the historical consequences might have been in the first test failed, lots more details on the "can you destroy the planet with a nuclear weapon" question, including a close analysis of their work on this during the Manhattan Project.