Several countries had small, exploratory nuclear programs during World War II, including Germany and Japan. But both of these nations decided by 1942 that they were not going to try and push to make actual weapons for use during WWII itself, both believing (not completely incorrectly) that it would take a tremendous effort and that their resources and manpower were best spent elsewhere. Both also concluded that this would be the same for other countries. By comparison, while the US was initially on a similar path, UK scientists concluded it would be much easier than previously thought, and convinced the US of this. This is what led the US to pivot from being a small research program and into a large-scale, military-industrial-governmental production program — the Manhattan Project — in mid-1942. It ended up being significantly more difficult and expensive than was anticipated, but by the time this was realized they were "all in" on the effort.

Which is to say, the US program was highly unusual — it was the only nation that actually was trying to build a nuclear weapon for use in World War II, and it entered into this effort somewhat under a misconception about how hard it would be.

"Was it a matter of time before Nazi Germany would have invented one?" This is a tricky question because plainly not, given that Nazi Germany ended as a state in May 1945, but if you mean, "if somehow the war had never ended, would they have gotten an atomic bomb?" The German program, by the time the European part of the war ended, had not yet quite managed to make a working nuclear reactor prototype, something the US had accomplished by December 1942. So they were considerably behind the US; they were years away from having a nuclear weapon, because, again, they were not really running a nuclear weapons development program. But if you imagine a hypothetical reality where the Nazis had won the war, it is reasonable to assume they would have eventually developed one. They were not ignorant of nuclear technology's possibilities, they just thought it would be a problem to be solved after WWII, not during it.