Germany was one of the great scientific powers in the early 20th century. France and England were the other two — the United States was not especially known for its indigenous scientific work, though it was starting to build up the infrastructure and talent necessary to really shine. Germany suffered two majors "brain drains" as a result of World War II.

The first was the emigration of the Jews, which happened after Hitler took power and started enforcing draconian anti-Semitic legislation. One of the first acts passed by Hitler essentially fired all Jewish university professors. Many of them emigrated shortly thereafter. They ended up in a wide variety of places — e.g. France, the UK, Sweden — but many of them eventually ended up in the United States, because the US was more open to their emigration than these other countries, and many US universities were taking advantage of the situation to suddenly prop up US science with top-flight European talent.

Theoretical physics was one of the over-represented areas for Jewish talent, in part because it was considered, prior to WWII, to be a relatively lower-status form of physics, in that it had no obvious industrial applications. As a result, it was considered a job that Jews could excel in; the top-flight non-Jewish Germans went into experimental physics or engineering, which were more highly regarded. Thus there is some irony that theoretical physics ended up being very important indeed with regards to the development of nuclear weapons.

One of the areas heavily funded by Hitler, in part because it fueled his imagination but also because it skirted around the limitations of the Versailles treaty, was rocketry. So this was an area where the Germans ending up excelling on a world scale. It was a bad military decision, ultimately, because the field could not be advanced to a militarily-useful realm over the course of the war (the V-2 was heads-and-tails above anything else prior to it, but it still didn't prove to be a useful weapon — it was militarily ineffective and horrendously costly). In the postwar, however, the United States and the USSR both realized that if you combined long-range rocketry with atomic weapons, you would have a serious weapon that would easily compensate for the accuracy problems of the rockets. So they both sought to acquire as much German talent in this area as possible. In the US case, Project Paperclip, they ended up basically acquiring the top engineers from the German V-2 programs and putting them to work almost immediately designing rockets that would carry nuclear payloads.

Now all of the above aside, I would say you do have a slight misconception. The vast majority of the people, even technical people, who worked on the US nuclear weapons programs and the US rocket programs were not Germans at all, and most were not Jewish. However the Jewish emigres, and some of their former students (e.g. Robert Oppenheimer), were very prominent in the program, and often more colorful characters. It is similarly easy to over-focus on the Germans in the rocket program, and ignore the many other workers who contributed.

But the general perception that mid-century US science and technology was heavily influenced by the emigrations before and after World War II is not an inaccurate one.