Early on, they were very worried. When the project was conceived it was thought that it wouldn't be all that hard to build an atomic bomb and that the Allies were far behind, because they didn't really start until 1942.

Over time, however, two things happened. One is that they realized it was much harder than they had anticipated — it cost way more than they had expected, it required way more people than they expected. Their estimates were off by a factor of four or five; that's pretty significant. Even the US was struggling to produce working atomic bombs within the constraints (both time and resources) of the war. So that's a good sign that Germany would probably also would have found it much harder than they had estimated as well.

The second thing is that after D-Day, the Allies sent teams of investigators (the Alsos mission) into liberated territories, and specifically looked for signs of a Nazi nuclear program. So they learned that while the Germans had talked to the French physicists about their reactor work, their questions didn't indicate that they were doing anything that large and that they were still pretty in the dark about fission research. They investigators were stunned at how little secrecy the correspondence they captured exhibited — it seemed to indicate they weren't really doing anything on a military scale. The more information the Allies got, the more they began to see that the German program was very modest by comparison, and not really a bomb program at all. It wasn't any one thing that changed their mind; but observations were adding up, and there was a gradual decrease in worry about the Nazis, culminating in a November 1944 conclusion by the scientific head of the Alsos mission that the German program did not exist. Of course, there were some who were a tiny bit worried all the way up to V-E day, but November '44 is generally the date at which people were pretty sure that the German program had never gotten off the ground.

The Alsos worked continued even after the war, and led to the capturing of the top German scientists, and the whole Farm Hall mission to really conclusively confirm what had been done in their program. Nobody let their guard down, really, and there was a lot of interest to find out exactly how far it had and hadn't gone. But that is a different sort of concern than the genuine fear that existed at the beginning, when some top scientists (like Arthur Compton) were actually predicting that the US would have to weather several atomic bomb strikes before it could retaliate.

I've written at some length about this here, talking about the sources involved.