This thread does a pretty good job of answering the first part of your question. /u/Frogbrooks is the one who gives the highest rated answer to that thread’s question.

Good question. GOOOOOOOD damn question.

There are lotsa arguments and /u/frogbrooks has already covered it from the historical angle.

One of the new and more exciting avenues for this research has actually come out of south arabian inscription evidence by Christian Robin, who's been producing fantastic books worth of stuff (mostly in French). The real puzzling thing is how exactly a european researcher got unfettered access to research PRE-Islamic religious evidence in places like Yemen where such research could cause... well problems.

In short, one of the arguments he makes (and I want to stress arguments, because the other positions have been covered in some of the links) is that in south Arabia (which is below the central Arabian, or Hijaz region that Mohammad was from) the prime religion was a vague sort of priestly judaizing monotheism. Meaning, they were Jewish of the pre-second temple type (but without a temple) but not of the rabbinic law type (the ones we think of as jews now). This evidence comes out of those brand new translations of inscriptions that had previously not been looked at. The inscription evidence at the same time points to the fact that polytheism as was presumed in the early islamic narratives had already disappeared (from at least south arabia) 100 years before the birth of Islam.

Now, you might wonder why this has any relevance to the Hijaz, which is where Mecca and Medina is. Well the relevance is that we don't have any inscriptional information from the very closed off Saudi Arabia as to the pre-Islamic era, so we're dealing with a black hole. So the South Arabian information could mean one of two things: (1) South Arabia was judaizing and not pagan JUST LIKE Central Arabia, or (2) South Arabia was judaizing and not pagan, BRINGING PRESSURE on Central Arabia.

I believe (2) is more likely for the Hijaz because the south was ruled under the explicitly Jewish Himyarite kingdom, but likely it would also have faced pressure from the Christian Roman Empire and its Ghassanid allies to the north. It would make sense to maintain some degree of religious/identity separation for political independence. However, it's likely these assorted Christian pressures, that gave Islam its interpretations on the divinity of Jesus.

So what does that mean for your question, what was Mohammad's religion? Well the answer is... it was likely part of a fluid mishmash.

You have priestly judaizing monotheistic pressure from the south (the presence of jewish tribes is already accepted in the historical accounts in nearby medina), christian pressure from the north (and possibly also the west from the Aksumite kingdom), and maybe, MAYBE, residual polytheistic aspects still current in Mecca. This means that Muhammad would've already been well aware of the long Abrahamic roots of what would become the religion of Islam, as well as have a foil in the guise of likely rapidly declining traditional polytheism for him to fight against.

There's a world of stuff involved in this question, and I can provide some sources for you if you want to read more, but a lot of it is french. So feel free to ask any follow up questions if you want any details and I'll try to provide what I can (I wouldn't say I'm an expert at early Islam, but I'd say I am decently informed about some of the pre-islamic and early islamic situation).