Short answer: not far at all. They weren't actually trying to make an atomic bomb. They had a very modest research reactor program, and even that did not succeed. That's it. All of the historical controversy over this is about why they weren't trying: Did they have moral qualms? (probably not.) Were they totally incompetent? (Probably not.) Did they lack the resources necessary to even begin this kind of work seriously? (Probably.)

All of the drama about the "Nazi atomic bomb" and the "race for the bomb" is something that is manufactured to make for exciting stories and make it sound like the US only barely won World War II (or, worst, to justify the US atomic bomb program). But historians have known for decades and decades that this is a false framing — arguably since 1945.

Longer answer: see this previous thread.

Happy to answer more questions on this.