Wikipedia gives a pretty good synopsis of Operation Downfall, which was the Allied plan for the invasion of the Japanese home islands if they failed to surrender following the bombings.

Basically the plan was divided into two component operations: Coronet and Olympic. Coronet's objective was to seize the southern third of the island of Kyushu. Once this was completed, the ground seized would be turned into a large air base and supply station for the final phase, Operation Olympic. Olympic entailed the invasion of the island of Honshu at the Kanto Plain and the seizure of Tokyo itself.

The Japanese, for their part, correctly predicted the Allied plans. They never really believed they could defeat the Allies, but rather focused on making the invasion so bloody as to force the Americans to seek a negotiated peace. To that end, they went to great lengths to fortify Kyushu in preparation for the invasion. Many tell stories of the civilian populace being given sharp sticks and rocks and being instructed to take as many foreigners with them as they could before they died. They also counted on having large numbers of kamikaze planes to fling en mass at the invasion fleet, which they believed would seriously damage the expeditionary force before it could land. The Japanese had no serious plans to defend Tokyo, and staked everything on being able to defeat the invasion in Kyushu. This is because while Kyushu is naturally rocky, rugged terrain ideal for defenders and trouble for American armor, the Tokyo and the surrounding Kanto plain were considered "good tank country" where Allied numbers and firepower would have full effect. Basically, they put everything they had into defending Kyushu and prayed it would be enough.

The casualty estimates for Allied planners varied wildly. Truman would later claim he was told to expect over a million American dead, even if the invasion was successful. This was probably a bit of an exaggeration, and the most recent scholarship on the subject that I've read argues that they really expected casualty figures that were somewhere in the range of the Normandy campaign.

TL;DR If the Japanese didn't surrender after Nagasaki, the largest invasion fleet in the history of mankind would have been launched towards the Japanese home islands.

Edit: typos