The Japanese leadership was split into two basic camps. One, represented by the military (who had a majority in the cabinet), believed that the Japanese could essentially out-last the Allies by out-bleeding them. At its most tactical this was a calculation about what democratic nations would be willing to tolerate in terms of bloodshed, at its most foolish it was a romantic idea about a "final battle" in which honor would prevail. Either way it mandated unyielding war.

The other, represented by the foreign minister, the Emperor, and several other "civilian" cabinet members, sought a way out of the war. Japan wasn't going to "win" by any military definition, they clearly saw. The best outcome in such a situation would be a negotiated surrender, a diplomatic end to the war. The Allies had insisted upon "unconditional surrender" as the only terms they would accept, which the Japanese high command as a whole saw as unacceptable, because it would open up the country to be totally disassembled, totally made over, totally destroyed, by the Allied administrators. At the very least they wanted a preservation of the imperial house to be guaranteed. At the most, they maybe thought they could "get away" with keeping some of their overseas possessions that they had now administered for many years.

Over the course of the spring and summer of 1945, the "peace" wing of the Japanese cabinet did try to engage the still-neutral USSR in discussions about whether or not they would be willing to serve as a mediator between Japan and the other Allies. This came to nothing, because the USSR had already planned to renounce their neutrality and declare war on Japan, and take territory in the process. The US was aware of the Japanese entreaties to the Soviets, and to their problems with unconditional surrender, because they had long since cracked Japanese diplomatic ciphers.

Such was the situation in the summer of 1945: parts of the Japanese cabinet were still pursuing a "diplomatic" end, parts were still dug into a militaristic position. None saw unconditional surrender as remotely acceptable. If the US invaded, it would be long and bloody, and perhaps give opportunities for both ends — perhaps the US would be willing to negotiate a surrender rather than continue it.

The atomic bomb going off over Hiroshima did get the attention of the cabinet, but it did not push them to surrender. Instead, it pushed them to investigate what had happened at Hiroshima, which took several days (since the bombing cut the city off from communications). While that was happening, the Soviets renounced their neutrality and invaded Manchuria. This hit the Japanese cabinet like a shock, because it fouled up both the "peace" and "war" camps. The military had run analysis years before on the effect of the USSR entering the war, and concluded that they would be destroyed by them: the Soviet Union would take over crucial Japanese holdings in China and Korea, cutting the island off from supplies, and Stalin was not squeamish with trading the blood of his soldiers for war victories. The peace camp had put all their hopes in the USSR remaining neutral and negotiating a surrender with the other Allies; these hopes were now totally dashed.

During the cabinet meeting on August 9th when these matters were being discussed, the news about the Nagasaki bombing was also delivered. It does not seem to have changed anyone's fundamental views on the situation.

In the end, the Japanese cabinet agreed to offer a conditional surrender to the US: they would keep their Emperor, but would otherwise accept the US terms. This was rejected by Truman and Byrnes, who insisted on unconditional surrender. The US followed this demand up with further conventional bombing, and within Japan there was an attempted coup of the Emperor by junior officers who felt their honor was offended by the condition surrender offer. All of this led Emperor Hirohito to intervene and offer up an unconditional surrender on his own. Such ended the war.

There are several take-aways here. One is that the atomic bombs, unlike in your question prompt, are not unambiguously the sole or even major cause of the final surrender. This is an issue of dispute among historians but pretty much everyone these days agrees that the Soviet invasion probably played at least as big a role as the Hiroshima attack, if not more of one, given the context of the cabinet's thinking. Nagasaki does not seem to have played any role at all in their thinking, and probably the result would have been the same if it had not occurred. Whether Hiroshima could also have been avoided is unclear. There were projections even at the time that said that even without the atomic bombs, and even without the Soviet invasion, Japan would have folded prior to a US land invasion. There is much unknown here, necessarily, but it is just worth pointing this out since it goes so strongly against the "orthodox" story that most people are taught (that the bombs ended the war, that without them there would have been an invasion, etc. — this "orthodox" narrative was created in the years after the atomic bombing by people who were involved in planning the bombing for the purpose of justifying the bombing).

Second, one can rightly ask whether or not the end of the war could have been brought about sooner if the unconditional surrender requirement had not been insisted upon. Certainly this is something many people suggested then (including people in Truman's cabinet, like Stimson, and other advisors to him, like Churchill) and later. Again, hypotheticals are hard to know, but it is an interesting point to consider, especially since people are rarely taught (as your question makes clear) what the Japanese point of view on this was. In any event, the US did decide — only after the war ended — that it would allow the imperial house to be preserved so long as it took a more ceremonial (and less religious and political) role in Japanese governance.

Third, and finally, whenever thinking about "success" it is important to make sure one understands what the goals were for the people at the time. Often people assume that the Japanese (or even German) goals were a vague "world conquest" that would mean invading the US or something like that. In reality their goals were large but not that large: they wanted to become the predominant Asian power, governing other Asian countries as puppets. Their goals toward the end of the war were more mundane: preserve the Emperor, preserve some of their holdings. They could not achieve those goals by accepting unconditional surrender, so it makes sense — at least, until the rug got pulled out from under them — that they would stay in it even if it appeared that their long-term goals were fruitless.

There are many books written about this period, but the most useful for understanding the Japanese perspective at the end of the war is Tsuyoshi Hasegawa, Racing the Enemy.