Was it understood that Mohammad and his followers were just some off-shoot heretical branch of Judaism/Christianity, or an entirely separate group? Was it clearly delineated early on? I know that the delineation between Jews and Christianity formed largely in part due to the violent reactions each group increasingly acted out against each other with, under the influences of the Roman legal system and society as well, but Islam grew out of a 'pagan' society. Were there Christian or Jewish societies in Mecca and Medina that contested the legitimacy of Islam as well?
Answers regarding the Ethiopian kingdoms are welcome as well, since I know Ethiopia had long had an interest in Abrahamic religions and at various times espoused Judaism or Christianity as its state religion before Islam came into existence.
This is a phenomenal question that truly deserves an in-depth answer, which is one that I'm not entirely sure I am capable of giving. However, what I can do, is at least give something of an overview of the controversy that has arisen over the ecumenical identity of the early Islamic community.
Traditional Muslim scholarship will generally suggest that Islam arose in an environment that was dominated by pagan polytheism. Traditional accounts of the Prophet's life will state that the majority of Mecca's inhabitants worshipped a variety of deities and that Mecca's economic prosperity at least partially depended on pilgrimage, which coincided with other economic activity. The goal of the pilgrimage was, of course, the Ka'aba, where various idols were worshipped. However, Muslim traditionalists will also allow for the existence of non-denominational monotheists, referred to as ḥanīf (pl. ḥunafā'); according to (nearly) all accounts, Muhammad himself was also a ḥanīf.
Furthermore, some of the sources also suggest a small Christian presence in Mecca. After Muhammad received his first revelation, it is said that his wife Khadija's cousin Waraqa b. Nawfal consulted him and reassured that he was not possessed, but had in fact received the Angel Gabriel. Waraqa b. Nawfal is described in the Muslim sources as a Christian monk. The tradition also describes the presence of several Jewish tribes in Medina (then still named Yathrib), who were amongst the ones that invited Muhammad to move there c. 621 AD. The conflicts that arose with these different tribes would, according to the tradition, have a significant influence on the development of Islam following them, including the "correction" of the qibla (direction of prayer) from Jerusalem to Mecca.
In short, from a traditional or classical Islamic perspective, Islam itself is the correction of the "original" Judaism and Christianity, before they came to be corrupted.
--------
However, we can problematize this view from a historical-critical position. As you may be aware, outside the Islamic tradition there is relatively scant information about Central Arabia of the 6th and 7th century. For example, while the city of Yathrib is attested in Akkadian and pre-Islamic South Arabian inscriptions as well as in Graeco-Roman geographical works, there are no solid references to Mecca. * In the past few decades or so, our understanding of the region has improved significantly, particularly due to the wealth of epigraphic evidence that continues to be discovered and deciphered, but regardless, the material is slim.
You are probably familiar with the central thesis of Patricia Crone's and Michael Cooke's Hagarism, which promoted a fairly revisionist interpretation of early Islam by functionally discarding the entire Muslim tradition and relying exclusively on non-Muslim sources contemporary with the rise of Islam. While their ideas were justly criticized and critiqued, and the original authors also distanced themselves from their original thesis, it did raise the question of how much we actually can know about Islam's earliest history. What is particularly problematic is how little evidence there is for sustained polytheism in western and southern Arabia during the 6th century, which led Crone to ask the question: "then who are the pagans (mušrikūn) that the Qur'ān denounces so strongly", and led to the conclusion that it was actually talking about trinitarian Christians (Crone 2010).
Anyway, someone like Fred Donner actually asked exactly the same question that you also posited. Donner promotes a view of the early Muslim community as being highly ecumenical in nature; he calls this group "the community of Believers", reflecting the Arabic term muʾmin (pl. muʾminūn). In fact, Donner explicitly states that the Qur'ān gives "no reason to think that the Believers viewed themselves as constituting a new or separate religious confession" (2010:69). The community of Believers was mostly distinguished from Christians, specifically, regarding their attitude to the Trinity (as mentioned before). It was only during the Umayyad Restoration that began with the reign of Abd al-Malik (685 – 705) the theological underpinnings separating Islam from Judaism and Christianity solidified. The clearest material evidence for this, according to Donner, is the construction of the Dome of the Rock, particularly in light of the Quranic inscription on its walls (19:33-35), specifically denouncing the notion of Christ's divinity.
This argument does seem to find some support in what little early material evidence we have. Whereas some of the earliest inscriptions mention the hijra (Muhammad's exile to Yathrib/Medina), they do not mention the Prophet by name nor by title. As Anthony points out, this is a rather puzzling observation. Apparently something so important happened in 622 AD that it resulted in the adoption of a new calendar, but references to the Prophet remain implicit. However, it might lend some credence to Donner's suggestion that the position of the Prophet as one of the central markers of Islam really only developed towards the end of the 1st to the beginning of the 2nd century AH.
If you are more interested in this question, I would strongly recommend reading Donner's Muhammad and the Believers – At the Origins of Islam. For more about religion in Central Arabia during late antiquity, see the chapter "Religion" in Hoyland's Arabia and the Arabs. For a critique of Donner's theory, Sinai's "The Unknown Known: Some Groundwork for Interpreting the Medinan Qur'an"
(* The importance of this observation is, I believe, overstated, but it might imply that Mecca was perhaps not as important in the pre-Islamic period as we would be led to believe).
----
Sources
Anthony, Sean W. 2020. Muhammad and the Empires of Faith: The Making of the Prophet of Islam. Oakland, California: University of California Press.
Donner, Fred. 2010. Muhammad and the Believers – at the origins of Islam. Harvard University Press.
Ibn Isḥāq, Muḥammad. 2001. The Life of Muhammad: A Translation of Isḥāq’s Sīrat Rasūl Allāh. edited by M. ʿAbd al-Malik Ibn Hišām. Karachi ; New York: Oxford University Press.
Hoyland, Robert. 2001. Arabia and the Arabs – From the Bronze Age to the Coming of Islam. New York, NY: Routledge.
Sinai, Nicolai. 2016. “The Known Unknown: Some Groundwork for Interpreting the Medinan Qur’an.” Mélanges de l’Université Saint-Joseph(66):47–96.
In addition to the excellent answer above by u/Kiviimar, a similar question was asked last year and prompted a number of differing responses representing pretty broad scholarly opinions: Previous thread here