First, popular opinion had little to do with it. The insistence on unconditional surrender was not done because of poll data. It was done because Truman and several of his top advisors (notably Byrnes) really wanted it that way. Though several of his other top advisors (notable Stimson, and even Churchill) thought that it was an unnecessary insistence (because they knew, through intercepted Japanese communications, that this was a major difficulty for the Japanese).

As for the "threat" — they posed no threat to the continental United States at that point. They could no longer expand for the near future. But that isn't how anyone framed it. The question isn't whether they continued to pose a short-term threat, but what the ideal state of the end of the war would be. With regards to Germany this was relatively clear: they wanted the Nazis removed, and they wanted conditions that would prevent that kind of thing from ever happening again. They appreciated that the end-of-war conditions for WWI had caused WWII, and so wanted something different from the Versailles treaty (where Germany was heavily penalized and left to suffer). So this ended up meaning dividing it into zones of occupation, de-Nazification, and eventually the Marshall Plan — rebuild them into an ally.

With Japan it was less clear, because Japanese imperialism was not a result of any single, obvious cause. They blamed the system of government in part for it — the ability of Japanese militarists to hijack policy. So they wanted them out for sure. They wanted Japan to eventually be some kind of stable liberal government, but whether that was a "true" democracy or a constitutional monarchy was less clear. They wanted Japan to be an ally to the West against the USSR, but also wanted to defuse the "Emperor death cult" that had led to World War II (in their eyes). And this involved rooting out and changing the system of government, which we didn't believe they would do on their own.

But a lot of the insistence on "unconditional surrender" also seemed to come from the gut — e.g. "you did this to us at Pearl Harbor, this is the humiliation you deserve." This is more or less my read on Truman's own personal views, which strike me as being based less on realpolitik than on a loose idea of vengeance, and maybe a perceived view that the American public would prefer this form of retributive "justice." The other options on the table involved something more "conditional," e.g. giving the Japanese strong assurances to how the Emperor system would be preserved or transformed. (As it was, we agreed to let them keep the Emperor as a figurehead, but neutered his political power.)

OK, separately there is the fallacy of bomb versus invasion — they were both on the table because obviously they didn't know if the atomic bombs would in fact end the war. (Or whether the declaration of war by the Soviets would end it — something that seems to have been in the air as a possibility, and was in part a justification for the timing of the bombing.) The biggest historical fallacy though is the idea that Truman or his advisors really agonized over this question, whether they had an elaborate calculus of saved versus taken lives that they were using. This is just not reflected in the documents of the time. They had an atomic bomb, they were going to use it. The questions they talked about were how to use it, not whether to use it. They had discussions about what kinds of targets to use it on, what kind of publicity/propaganda should be released after using the bomb, whether there should be a warning or demonstration about the bomb, and so forth. But the actual use of it, in one way or another, was mostly taken for granted. In the postwar, when the bombings became controversial (esp. around 1947), this elaborate narrative of the "decision to use the bomb," with its moral debates and whatnot, was created, but this was a very after-the-fact justification, not reflective of how they were actually thinking about it.

It seems like the best solution was to not invade and not drop the a-bombs. So many lives could have been saved.

In many ways, the Japanese stopped being a threat to us by 1944. The demand for the unconditional surrender of Japan was, in effect, a way to prevent them from precipitating another war somewhere down the line by removing the tools of aggression from their hands.

There were three realistic options available to end the war. First, the bombs. Second, blockade. And third, invasion.

We know what the bombs did: around 150000 dead, more or less.

Blockade would have saved many, many lives, almost all of them American. The civilian population of Japan would have been decimated by malnutrition and starvation. As it was, the bombing of Japan that had already occurred ruined much of the infrastructure of the nation. Japan needed a massive influx of food from the US to prevent famine immediately post-war, a shortage that lasted for a few years. The death toll from that would easily range into the millions of civilians.

Invasion would have a similar death toll from famine, plus combat losses from the military on both sides. Japanese combat losses just for the invasion of Kyushu would probably be near a million troops and civilian auxiliaries. and vastly more for the whole overall invasion of Japan. Allied losses have been constantly debated and confused... you can find anything from 100k deaths for the invasion of Kyushu to half a million to a million for the overall invasion of Japan.

Dropping the bombs of Hiroshima and Nagasaki saved lives. They were the best of a bad series of choices.

(please note that none of this includes the possibility of Soviet participation... best not to muddy the waters too much)

Useful Sources:

Downfall - Richard B Frank

Hell to Pay - DM Giangreco

The Invasion of Japan - Richard Skates

Embracing Defeat - John Dower

"Olympic vs Ketsu-go" - Bauer/Coox, Found here

"Operation Downfall" - the original strategic plan for the invasion of Japan