I wouldn't call it a small movement. The US officially released a disarmament plan to the United Nations, in all seriousness. The idea was to come up with a UN-run system that would eliminate nuclear weapons globally, preventing any nation from developing them. (With some possibility about having the UN itself maintain some nukes, as a hedge.) This is the Baruch Plan, which itself was derived from the Acheson-Lilienthal Report, which itself was written with a lot of input from David Lilienthal — future Chairman of the US Atomic Energy Commission, the postwar organization that produced nuclear weapons in the US — and J. Robert Oppenheimer — the "father of the atomic bomb." Which is to say, none of these characters were fringe, and this was official US policy.

It didn't go anywhere, because the Soviets were not that interested in it (they didn't trust the US to do it, and the plan involved basically locking down the world's capacity to produce nuclear weapons before the US would give up its own stockpile). But between the end of 1945 and about 1948, this was a serious proposal and debated at the highest levels of international governance.

At the very end of 1945, a national poll showed only 14% of Americans in favor of this kind of international control, and 73% against. But another poll in 1948 showed 88% in favor of trying it, with only 12% dead-set against it. Polls from in between these days show a shifting in favor, though it strongly depends on how the questions are asked (e.g., whether it is said explicitly that this would involve the US destroying its existing bombs). Which is just to say, it wasn't politically impossible — you could imagine selling this to the US populace if it seemed promising.

The fact that the Soviets were building their own bomb, and would detonate on in August 1949, and seal into place the arms race of the Cold War, makes all of this a remarkable thing to contemplate, but it's one of my favorite historical episodes, because it is only with the benefit of hindsight that you can say this whole effort was doomed. At the time, it could have gone either way — and with it, history might have gone on a very different (maybe better, maybe worse) track.

For details on the international control campaign, see Alice Kimball Smith, A Peril and A Hope, and Michael Gordin's Red Cloud at Dawn also covers this well.