First, be aware that while $25-30 billion sounds tame today, it was a big deal for World War II-era spending. (We have just gotten used to dropping a trillion here and a trillion there, as if it is no real money at all.) The Manhattan Project cost the same amount as the entire B-29 program — it was a major expenditure. The number that reoccurs in many Manhattan Project statistics (spending, labor, number of patents) is "about 1%" — it was about 1% of total WWII spending, used about 1% of total US civilian labor, and was responsible for about 1% of the total US patent applications developed during WWII. (It also used on the order of 1% of all US electricity!) 1% may not sound like a lot but consider how large the US economy is, how large the WWII war effort was. It was a huge expenditure for a singular research and development program.

As for how they did it, the answer is not extremely surprising. They got someone to lead it who was willing to do essentially anything to make it succeed (Leslie Groves). They gave his project top priority for the entire war — if he wanted people, resources, whatever, he got them. They insulated the project from all oversight and inference (making it essentially undemocratic), and ran it as a virtually autonomous shadow empire who answered to almost nobody lower than the Chief of Staff of the Army. They then allowed him to hire as many technical experts as he wanted, including many of the top scientists in the world. And they allowed him to contract with the largest industrial firms in the country — Westinghouse, DuPont, etc. — to develop individual plants. And — and this is key — they allowed him to use virtually unlimited funds and labor for one of the most productive countries in the world.

As a result, the Manhattan Project pursued multiple full-scale avenues towards the bomb simultaneously and without any real review, oversight, checks, anything. (It's a terrible model for future policy as a result, as an aside, and I bristle every time I hear someone call for a new "Manhattan Project for X" — maybe you think the Manhattan Project had great outcomes, which is debatable, but its way of getting them is exactly the opposite of what you should want out of a government spending program.) It did things that in any other time would be considered very, very dangerous. I mean, creating a nuclear reactor under a squash court in one of the great urban cities is not exactly sound practice. Nor is immediately scaling that up to industrial size in one step. Nor is creating weapons that had a fairly high chance of detonating should the plane carrying them crash (the early weapons were "hideously dangerous" as one of their designers later put it). Nor is storing copious nuclear waste in shoddy underground vats with the hope that someone would soon clean it up (we're still cleaning it up). Nor is creating a polonium-production facility in a suburban neighborhood playhouse. Etcetera, etcetera. The Manhattan Project model was an totally unsustainable model of development.

One can ask why other nations have not beaten the Manhattan Project speed record. The Soviets, as you note, took about 4 years, even with espionage. To answer this question requires looking at what set the pace of the Soviet program. The answer wasn't information — the stuff of espionage — which did not set the pace of the US program, either. The answer was the acquisition of uranium resources and turning them into fissile materials. So this points to another advantage of the US project: it had virtually unhindered access to huge amounts of raw uranium, largely from the Belgian Congo but also from US and Canadian sources. The Soviets did not; they were extremely uranium-poor at the end of World War II and had to expend considerable effort locating uranium resources and then use copious labor to extract them. Their uranium sources were also very poor, requiring more effort than the US required to make them usable (the Congolese ore was several orders of magnitude more rich in uranium than Soviet ores were). They also, it is worth noting, did not use the espionage as a way to "speed up" — they used it primarily as a way to check the work of their own scientists, to make sure they were doing something that would for-sure work. That is a very different management style than the Manhattan Project, whose leaders had no doubts about the integrity of their own employees. (There is a difference, in the end, between a Groves and a Beria.)

We might look at other nations as well. The British were in far less of a hurry to develop their bomb and so they took a path that would be far cheaper in terms of labor and resources. That can work, but only at the cost of time. The French were in a similar situation.

The Chinese are of interest because they proceeded as fast as they could, but were actually hampered by partial assistance from the Soviets, which, once withdrawn, caused them to make many mis-steps and then, finally, re-learn everything from the ground up. They ended up with a very "fast" program (they are the fastest transition from fission to thermonuclear weapons), but only after many hard lessons learned.

The post-NPT nuclear powers in general developed their arsenals under conditions of inspection and subterfuge and that entails its own costs in terms of time. When you are being actively examined and scrutinized, you cannot do things the Manhattan Project way — you cannot draft every capable scientist in the country onto the project, you cannot construct the largest factories in the world at the time, you cannot mobilize 1% of your workforce. People will notice. So these programs are notable both by their secrecy (India in particular) but also by the fact that these nations generally had a lot fewer resources to spend (compare the GDP of North Korea or Pakistan to the US in WWII — orders of magnitude differences). These smaller, cheaper programs got results, but only at the expense of a lot of time. And many of them were poorly managed; there was no General Groves, no hard-charging, make-it-or-be-damned leader who was autonomous from the political structure.

So the Manhattan Project had many things going for it, even in the face of the uncertainty of not knowing which of its methods would prove most successful, if any. I would also just note that the Manhattan Project leaders also just got lucky in some respects. They were on a razor-thin margin between success and failure, and they knew it. Groves brooded frequently on the consequences that would result from not having a weapon ready to use in the war, or a weapon that did not deliver on the results promised. He repeatedly joked that he would be spending the rest of his life giving Congressional testimony if he didn't pull it off. He feared Congress more than the Soviets and the Germans — only Congress could shut down his program prematurely. If they had been a few months late (an easy thing to imagine), the war could easily have ended (one way or another) without the atomic bomb. Which is only to say: one should not necessarily see the Manhattan Project's success as a foregone conclusion, and its "effectiveness" is in part an artifact of the fact that it worked, which was a very chancy endeavor.