I have only seen a few discussions of this in the historical record, mostly from the phase when they were just starting to think about what specifically they were going to do with the bombs once they had them and what they were going to do after they were used (they started this sort of planning in late 1944, and it picked up speed in early 1945).

For example, in September 1944, James Conant and Vannevar Bush wrote to the Secretary of War urging that he start thinking about what would be done right after the bombs were used. This included matters like coming up with press releases, model legislation to introduce into Congress, and treaties with the UK and Canada. They especially emphasized that long-term secrecy was unlikely to be feasible. The wrote:

(1) If atomic bombs are used against the enemy, as we believe they will be if the Japanese war continues for another year, it will be essential to release to the public information about these bombs simultaneously with their use. This information, in our opinion, should be presented in the form of a rather detailed history of the development which gives all the essential scientific facts and assigns credit to the many individual scientists concerned. [Note: This was the germ of what became the Smyth Report.] At the same time, it would seem appropriate for the President to report to Congress in more general terms and to propose a treaty with Great Britain and Canada for ratification by the Senate, as well as to introduce legislation to control the production of atomic energy.

(2) If the war with Japan should come to a conclusion before atomic bombs were used against the enemy, we believe that immediately at the close of the war the information mentioned in Section (1) should be released in the same manner. We cannot emphasize too strongly the fact that it will be quite impossible to hold the essential knowledge of these developments secret once the war is over. There are far too many people involved in the whole undertaking and too many individual concerns; only the excellent cooperation of the press on a voluntary censorship basis has prevented the subject being widely discussed in print. Furthermore, we should like to emphasize that the progress of this art and science is bound to be so rapid in the next five years in some countries that it would be extremely dangerous for this government to assume that by holding secret its present knowledge we should be secure. We advocate the fullest disclosure of the essential scientific facts but not the blue prints of the manufacturing processes or the construction of the bombs, either at the time the atomic bomb is used, or if this has not occurred before the close of the Japanese war, at the time of the surrender of Japan.

So you can see that this is sort of a passing notion, but not an insubstantial one, and connected to the question of whether the US should try and keep the whole thing secret indefinitely (which they think is a bad idea).

This context — release of information, etc. — is about the only one I've seen this issue brought up in. This does not mean it did not worry them. On the contrary, General Groves seems to have believed that if the bombs were not used before the war ended, it would mean his hide on a plate for wasting resources. As it was, it was a pretty close shave. The Japanese were making peace feelers in the summer of 1945, and it is not far-fetched that the war might have ended (on some terms) prior to an American invasion. If the Manhattan Project had been delayed by three or four months (which, as anyone who is gotten behind on deadlines knows, is not that much time), they might have ended up in a situation where the war ended before the bombs were ready.