No one can definitively answer this question. The planned invasion of the Japanese home islands, codenamed Operation Downfall, consisted of two separate landings. Olympic would involve the landing of 432,000 US troops in the first two weeks from X-Day on the island of Kyushu and was scheduled for November 1st, 1945. Operation Coronet would be the follow up invasion of the Tokyo plain the following Spring.

The question of whether the casualties for an invasion of Japan would have been larger than the deaths from the two atomic bombs revolves more around whether the invasion would have taken place as scheduled, and when Japan would have finally surrendered. There is little doubt that an actual invasion of Japan would have resulted in more deaths.

The invasion of Okinawa resulted in the deaths of over 100,000 Ryukyuans, nearly one quarter of the population. Kyushu had 10 million civilians, who had been fed propaganda for years that the Americans would rape and murder them if they were captured. 100,000 Filipino civilians were killed in the battle of Manila alone (although the Japanese military certainly wouldn't execute civilians during an invasion of Kyushu like they did in the Philippines.)

On the military side, the Japanese had stripped all but one of the front line divisions from the Kwantung army and raised several other garrison divisions on Kyushu. This meant that the eventual 600,000 US troops would actually be outnumbered on the island against 900,000 Japanese. Throughout the US island hopping campaign, battles resulted in nearly 95% KIA rates before the end of fighting, which even with a US victory would put the death rate much higher than the numbers killed in the atomic bomb.

The US was slowly realizing through signal intelligence that they were at least meeting an equal size force on Kyushu that threw the who invasion in doubt. The Navy, which had suffered horrible casualties (~4900 deaths) from kamikaze attacks at Okinawa was already opposed to the invasion prior to this discovery. The army felt that time was the enemy, and supported an invasion. MacArthur certainly relished the idea of leading the largest amphibious invasion in history. The Navy felt that casualties were the enemy and that the blockade would be successful.

The real question is how quickly would Japan have surrendered. Even without an invasion the civilian death toll due to starvation and exposure would have been horrific if the war had continued into 1946 due to the loss of food supplies from Manchuria and Hokkaido (the shipping and rail to deliver food from Hokkaido had been destroyed by August.) Japanese civilians were already being directed to eat acorn husks and similar items to supplement the near starvation diet that they were forced to live on.

Sources:

Drea, Edward J. MacArthur's Ultra: Codebreaking and the War Against Japan 1942-1945.

Frank, Richard B. Downfall: The End of the Japanese Empire

Assuming that an invasion went ahead on the scheduled date, 1st November 1945 (which assumes that Japan had not surrendered in response to the Soviet invasion of Manchuria, which would probably have continued to a Soviet invasion of Korea, and possibly a Soviet invasion of Hokkaido), we can estimate possible deaths.

Judging by the Japanese deaths during the invasion of Manchuria, when Japanese units fought had, but tended to surrender before complete destruction, there would have been about 300,000 more Japanese military deaths in Manchuria. A Soviet invasion of Korea could have been a foretaste of the defense of Kyushu if the Japanese-organised militia units - civilians armed with whatever weapons could be obtained, including bamboo spears, bows, swords, and ancient muzzle-loading guns - fought in significant numbers. Whether this would have happened isn't clear - if the Japanese had continued to fight in Manchuria, much of the Japanese army in Korea would have crossed the Yalu and been consumed in the Manchurian fighting.

US estimates of casualties for the invasion of Kyushu are unreliable, since they were based on a very low estimate of the Japanese strength on Kyushu (about twice as high as the estimates when Japan surrendered, and would have been higher by the time of invasion). The US estimates varied, from about 70,000 to 500,000. About 200-250,000 is typical of the "better" estimates, which would have included about 50,000 dead. Japanese military casualties on Kyushu could have easily exceeded 400,000.

To this would need to be added further civilian casualties in Japan due to continued conventional bombing, civilian deaths in the defense of Kyushu and Korea, which are difficult to estimate (in principle, extrapolating from the experience of Okinawa and Saipan, about 1/3 of the civilian population could be expected to die, either fighting or through "suicide" (given the extensive "assistance" to suicide on both islands by both Japanese/Okinawan soldiers and civilians, some of those "suicides" were probably not voluntary, and would be better classified as murders)). Conservatively, one could estimate 1-2 million civilian deaths on Kyushu due to the fighting, whether the civilians were resisting as ordered (with bamboo spears, etc.), were killed incidentally, or committed suicide. Strong civilian resistance could have increased Allied casualties.

The fighting on Kyushu was expected to take at least about 4 months, possibly with a shift to defence by the Allied forces, rather than continuing to advance, after 120 days. This means that, assuming that Japan continued to fight, to about 4 months after the invasion, the surrender would have occurred about 6.5 months later than it actually happened. The Japanese occupation of much of East and SE Asia, and the fighting over it, was killing about 250,000 people per month. This delay in surrender would have probably resulted in the deaths of about 1.5-2 million people in occupied Asia.

Therefore, assuming that Japan did not surrender until an invasion of Kyushu, and after the fighting there was mostly mover, about 4 million more people would have been killed due to the war. This is many more than were killed in the atomic bombing, and about half of them non-Japanese. If Japan did not surrender at this point, then there would be further deaths from the invasion of Honshu, and many more deaths in occupied Asia.

The big unknown is when Japan would have surrendered in the absence of the atomic bombings. The Soviet invasion of Manchuria was a huge shock to the Japanese (they weren't expecting it until 1946), and accelerated the surrender. Given that the Japanese cabinet was deadlocked on surrender even after the atomic bombings and the Soviet entry into the war against Japan, it is likely that without the combination of both shocking events, surrender would have taken longer. It is quite possible that Japan would have surrendered before invasion, but as noted above, the Japanese occupation of parts of Asia killed many people.

Comparing Japanese deaths due to the atomic bombings with the total deaths due to an invasion and a 6.5 month delay in surrender is very much a post-facto thing - it isn't something that would have been considered at the time, and was not a factor in the decision to drop the atomic bombs. That the atomic bombings appear to have been a lower-death option is simply that war is dangerous, and finishing wars sooner tends to kill fewer people.

While I do not have the citation at hand (we read it way back in grad school seminar), I remember that at least one draft of the Navy for the landing proposed a "clearing of the beaches" by dropping several nuclear bombs (requiring the landing to fall into '46 in any case, given the nuke production rate) to then land the US troops on these "cleared" beaches in what would have been irradiated zones.
While there is no way to know whether this would have been done, the plan alone shows several things:
- there was no understanding of nuclear fallout in military planning in the summer of '45; zero, zilch, nada;

- US casualties during landing were expected to be bad enough, and if this plan had actually been implemented, subsequently even worse;

- there was no implicit understanding that the usage of the bomb would actually end the war.

The latter point is underlined by the fact that, in a cover letter for a potential public announcement that the US had used a nuclear bomb for the first time, Harvey Bundy (a trained Harvard lawyer) wrote to the President in early January '45:
"This is the first draft of a speech to be given by you when S-1 is being used." [S1 being the bomb].
Given that a Harvard lawyer instinctively knows the crucial difference between "if" and "when", it shows that there was absolutely no expectation of an "if" debate to use the bomb. It was a new weapon developed at extremely high cost, so Stimson and Bundy expected it to be used like any other new weapon developed in the war, whenever it was ready.

Any claim that a) the bomb was dropped to end the war, and that b) it was a morally "better" choice made at the time, is projection from hindsight: The navy even planned to use it for the landing.
Any debate that discusses the morality of dropping the bomb or not and claims it was a choice made at the time is, in my view, pretty much moot.
Furthermore, to my knowledge, there is a complete absence of sources from before the August 6 & 8 usage of the bomb(s) that discusses an if in the White House or military: Stimsons' diary was edited before publishing; Truman's memoirs were based on short notes, and written out only afterwards and published later, when a debate about the morality of dropping nuclear bombs on largely civilian targets had begun. In the light of that particular political situation, no politician wanted to look as if they had not thought the consequences through; yet, if they actually did, where is the paper trail?

You've gotten a lot of good answers here about why it's hard to say one way or another about the loss of life in a possible invasion, but one thing that I don't really feel like these answers have brought to the forefront is that "bomb or invade" is very much of a post-war construction. There was no either/or on the part of American military planners during the war -- if anything, the use of the atomic bombs was overdetermined.

Truman came into office knowing nothing about the atomic bombs, and only made two main decisions about them: one, to take Kyoto off the "reserved" targets list for the bombing (more on that in a minute); and two, to stop the atomic bombings after Nagasaki. The popular image of him agonizing over using the bombs versus authorizing an invasion has no basis in reality; he was told that they were planned to be used and went along with it until the second bomb was dropped, whereupon he halted further bombings without his personal approval. (It seems to have bothered him that Nagasaki was a city; it's not clear whether he realized Hiroshima was one as well.)

In August of 1945, the U.S. and Britain both had large fleets that were busily attacking Japanese warships, harbors and shipping; American submarines continued sinking Japanese merchant fleet vessels in large numbers; aerial mines were being used to blockade harbors; and Army B-29s were dropping thousands of tons of napalm on Japanese cities; 67 cities were bombed before Hiroshima, with civilian casualties in Tokyo alone exceeding 100,000. The "reserve" target list I alluded to before was meant to keep some cities undamaged so that Manhattan Project scientists could study the effects of the atomic bombs on them.

I bring this up not to make any moral judgment about the use of the bombs themselves, but simply to point out that the use of aerial weapons on civilians was a line that the Allies had crossed very much earlier in the war, first in Europe and then on the Japanese home islands once they were within B-29 range. The first two atomic bombs were used mostly because they were available at the time; the Nagasaki bombing (originally scheduled for Kokura) was actually moved up to be closer to the Hiroshima attack due to bad weather in the forecast, but the plan was to keep using bombs as they became available, even as supports for invasion forces.

The framing of "bomb or invade" also rests on the assumption that the atomic bombings forced Japan to surrender, which is very much a debatable topic among historians -- the bombings coincided with the Soviet Union declaring war on Japan, and reasonable people can and do assign causality to the bombs, or the invasion, or both, in pushing the country finally into surrender.

Lots of sources on this, but much of it is covered in this FAQ section. If you read just one post out of there, make it this one.