How was Mohammad, an illiterate merchant, able to go into a cave and memorize an entire book by heart?

That's not the religious account of how Muhammad's revelation proceeded. Muhammad's first revelation in the cave at Mount Hira was short:

Read: In the name of thy Lord Who createth,

Createth man from a clot.

Read: And thy Lord is the Most Bounteous,

Who teacheth by the pen,

Teacheth man that which he knew not.

As indeed it is taken that the primary "units of revelation" were indeed short. This was revealed to him at the age of 40 (i.e. in 610 AD). There is greater disagreement about Muhammad's last revelation, which is often believed to be verse 5.3 which was revealed at the end of the Hajj in 632, three months before Muhammad died:

This day are those who disbelieve in despair of (ever harming) your religion; so fear them not, fear Me! This day have I perfected your religion for you and completed My favour unto you, and have chosen for you as religion al-Islam.

But there are other traditions that other verses of the Qur'an were the last revelation. In any event, the Qur'an came to the prophet Muhammad over the course of some 22 years.

an illiterate merchant, able to go into a cave and memorize an entire book by heart?

All of the above being said, if we wanted a secular explanation for the Qur'an, Muhammad's supposed illiteracy would not really be a problem. This was an oral culture. All of the poetry and literature of pre-Islamic Arabic was composed, recited and memorized orally. Illiteracy was no bar to the composition of literature of the highest order.

I also say "Muhammad's supposed illiteracy" because the fact is that most theories of how the Qur'an came to be composed require it to have been transmitted in writing from a very early period in all likelihood during the lifetime of Muhammad himself and there is a good degree of skepticism about the extent of this illiteracy.

is this the dominant theory for his encounter with the angel Gabriel?

It's not one I've come across and as I've said it doesn't make sense given that he continued to receive revelations until the end of his life.

I'm not aware of historians being all that troubled by accounts of religious revelation or religious experience. Accounts of extraordinary religious experience are not, historically speaking, all that extraordinary. Going back to William James' The Variety of Religious Experience if not earlier I don't think there has been much of a need to attribute profound religious experience to any environmental or other exogenous causes.

Is there any evidence he had ulterior motivations for declaring himself prophet, or did he sincerely believe he was a vessel of god?

As with the problem of Muhammad's illiteracy, the problem here is that our only sources for Muhammad's life and biography come from the Islamic tradition itself. This entire tradition, including the hadith and the sira, is tendentious and there are those who would do away with it entirely as being wholly unreliable. I would have a read of Robert Hoyland's Writing the Biography of the Prophet Muhammad: Problems and Solutions which can be found here.

There are actually, I think, more interesting questions from a "secular" perspective than the ones you've posed. For instance, what, if any, prior source material may have been used in the composition of the Qur'an? How was the Qur'an collected and what variant readings were redacted? What can we surmise about the changing nature of what the Qur'an was and what developments of thought occurred from the earliest (more poetic) verses to the later (more prosaic and legalistic) verses.

For further reading I would check out:

Neal Robinson's Discovering the Qur'an

and The Qur'an in its Historical Context which is a collection of essays.

edit: typo