There are a few categories of people whose health was impacted by nuclear testing.

The first are people who were involved with the tests themselves. These included on-site workers at the test sites, as well as soldiers who participated in simulated nuclear testing operations. Both of these groups were often exposed to higher-than-safe levels of radiation or fallout at times. This is a relatively small category of people though sometimes the numbers of soldiers could be quite large.

The second are people who lived directly down-wind of the test sites and were exposed to local fallout. Local fallout occurs when dirt and debris is mixed into the highly-radioactive nuclear fireball during the surface or near-surface burst of a nuclear weapon. This causes the radioactive particles in the cloud to descend back to the ground ("fall out" of the cloud) within a short amount of time (over the course of several hours), when they are still highly-radioactive and not yet diffused widely. In the United States, people living east of the Nevada Test Site (e.g. St. George, Utah) were repeatedly exposed to low levels of nuclear fallout, and many Marshallese in the Pacific Test Site were also exposed to low-to-medium levels of fallout. These groups are sometimes known as "down-winders." This was a medium-to-large group of people. In at least one instance a group of these people (in the USSR) was exposed to prompt effects of the blast as well (the blast wave reflected off of an inversion layer in the atmosphere, extending its distance dramatically, and collapsing a house in a nearby settlement, killing a child who was in it).

Lastly, there are the people who were exposed to global fallout. Global fallout is when the radioactive cloud diffuses into the atmosphere and much later settles back to the ground. It is not as radioactive as local fallout, but it covers a much greater area. Certain long-lived (low-radiation) isotopes are chemically similar to other elements used in ecological and biological systems and can end up in the food chain. Strontium-90, for example, is chemically similar to calcium and so shows up where calcium shows up (e.g. cow's milk) and from there can get into human food and thus bones. Millions of people were exposed to very low levels of radiation through global fallout.

The long-term health problems for each group can vary but in general the best way to think about this is epidemiologically. Radiation exposure increases the chance (risk) of getting various types of cancers. This risk is generally considered cumulative — even low levels of radiation increase this risk. In many cases this risk is very low for individuals, but when you increase the number of people exposed, you increase the total number of cancers in the population. So in the case of global fallout, the total number exposed was tremendous, but the total number of cancers is only going to be a small percentage of that. Still, a small percentage of a big number is a lot of individuals.

Hard numbers are difficult to come by and are necessarily estimates — it is impossible to trace back any particular cancer to any particular exposure source, but it is possible to see, epidemiologically, where "hot spots" show up and correlate them with exposures. Here's one set of estimates for the US program:

"As a result of fallout from U.S. atmospheric testing between 1945 and 1963, an estimated 70,000 to 800,000 people in the United States and around the world have died or will die prematurely from a fatal cancer attributable to the testing (a comparable number of fatalities would be attributable to the Soviet testing program)."

This is from Arjun Makhijani and Stephen I. Schwartz, "Victims of the Bomb," in Stephen I. Schwartz, ed., Atomic Audit, 395-431, on 395, based on "A Report of the IIPNW International Commission to Investigate the Health and Environmental Effects of Nuclear Weapons Production," and Radioactive Heaven and Earth. The authors estimate 430,000 fatal cancers owing to radiation exposure, about a third of these due to US atmospheric testing. (428) They clarify that if you run through carbon-14 exposures through infinity yield you get around 2.4 million expected cancer fatalities, 800,000 attributable to US atmospheric testing. All of this based on a linear, no-threshold hypothesis of radiation exposure for low doses. 70,000 figure is the low end of assuming that low doses incur cancer risk at a factor of 2 less than sudden doses, which is a common regulatory practice.

As for on-site and other categories, one can get some rough estimate of the US case from the claims filed under the Radiation Exposure and Compensation Act (RECA). As of this writing, some 22,700 claims have been filed by downwinders, and some 6,800 by onsite participants (or their relatives). These numbers may not be anything like a complete assessment and some of the claims may be erroneous (and many have been denied) but they give something like an order of magnitude estimate of the numbers involved.

I would just add that even radiation experts do not completely agree about the results of different types of long-term radiation exposure on populations, so none of these numbers are without controversy. A lot of our understanding about the relationship between dose and cancer comes from the victims at Hiroshima, Nagasaki, and the 1954 Castle Bravo test.