We don't know and almost certainly never will. There is no recorded contemporary history of the life and times of the Prophet Muhammad. In terms of documentary evidence the most reliable thing we've got is the Qur'an, which is not a narrative history in any sense, unlike the gospels.

I'm unaware of there being any suggestion that early Islam involved use of the Torah. There are a series of historical narratives called the "Israiliyyat" that cover the lives of the Jewish prophets but they are not portions of the Torah.

I think where part of this idea might be coming from is one particular work which was published a few decades ago called "Hagarism" by Michael Cook and Patricia Crone, rejected all traditional sources and came to the conclusion that Islam basically as an offshoot of Messianic Judaism.

The approach that Cook and Crone took has been hugely influential as later authors have sought to use as much non-narrative source material to construct a history of early Islam as possible. Their conclusion, however has been rejected even by the authors themselves. My understanding is that it has started to pop up in some popular histories without much context, however, and presented as a mainstream view. I'm currently only part way through the book but based on his TV documentary it seems that Tom Holland's "In the Shadow of the Sword" relies on this thesis quite a bit. But to reiterate, it doesn't really have much/any support.

There are some traditional sources that indicate a potentially different understanding of the communal relation between the faiths, most notably that the direction of prayer was apparently changed from facing Jerusalem to facing the Kaaba at Mecca (see the article on the Kibla in the encyclopaedia of Islam.)

Terminologically the Quran does refer to prophets of Judaism, Christianity and the new revelation as "muslims," i.e. those who submit to God so there is some degree of reliability regarding that.

Despite that the role of Muhammad as prophet and of the new religion in the early conquest period is likewise controversial as his name and prophethood don't turn up in official documents until surprisingly late. But when they do turn up there doesn't seem to be much question that this is a new faith with a new prophet, nonetheless it's a polarizing issue in the field, particularly because there is so little available material. Much of the most important work has come down to scrutinizing coinage.

Robert Hoyland's "Writing the Biography of the Prophet Muhammad: Problems and Solutions" covers all of these historiographic issues in some detail.

It is often said that followers of Islam also use parts of the Torah

Yes and no.

Muslims do not adhere to the Torah and are not required by faith to do anything with it.

However, on a secular and textual level, the Qur'an has the majority of it's parts lifted from the Torah. Muhammad was in contact with his wife's cousin, Waraqah, prior to his declaration of prophethood. Waraqah was a Christian and taught Muhammad about Christianity and Judaism (through the eyes of a Christian).

Christians and Jews were said to be among the different religious groups in Mecca, and Muslims do not deny that Muhammad heard parts of the New and Old Testaments from them or from Waraqah. By tradition Muhammad was said to be unlettered so he could not fully read those religious texts and thus be unable to compose the Qur'an without help from Allah.

But during the proto-Islamic period what "religion" would we consider the Prophet Muhammad and his followers? Were they Jews? Were they Christians? Were they something in-between? I have always been curious.

I can't tell you exactly when, but the Qur'an uses the word "Islam" many times to describe the religion. This website tells you the chronological order of the Qur'an, and I tried to cross-reference it with a Qur'an website but I couldn't find exactly when the Qur'an uses the word "Islam." It never uses the words "Christian" or "Jews" to refer to Muhammads' followers.

So around the time Muhammad declared his prophethood, he would have been regularly referring to himself and his followers as Muslims, and his religion as Islam. There was never a point, at least in Islamic tradition, where he viewed himself and his followers as a sect of Judaism or a sect of Christianity.