I'm not sure exactly where to date this. But say roughly from the High to Late Middle Ages. When Islam had fully absorbed the population of its conquests from Spain to India. But Christian Europe had not yet complete the Reconquesta or begun the massive expansion to the New World. So roughly the peak of Islamic expansion at the expense of the Christian World.
Did the Islamic population exceed the world Christian population during this time period?
This is my first attempt to answer a question! Its a big one to try, hopefully someone else can give more specifics. My quick search of the period you are asking about seems like 1000 C.E. to the introduction of the potato, which exploded the European population capacity. This is a complicated question because it is difficult to determine the rate of population conversion to Islam after the initial Islamic conquests of the Middle East and North Africa.
In a lot of cases, the best guesses are based off of the rising occurrence of "Islamic" names in the historical records. In Egypt for example, its not until the Mamluk Period, several centuries after the Islamic conquest of Egypt, that Copts finally become the noticeable minority based on name prevalence, but it could have happened during the Fatimids, its a bit of a grey area. If we look at other major population centers, there is usually an even more diverse mixture of religious groups. In Iberia there were many Jews and Christians.
It is also debatable of the origin of the peoples who migrated. There were certainly Arabs and Bedouin tribes that were part of the initial waves of conquests that made it to North Africa and Khorasan, from where the Abbasids would originate. And the Umayyads in Iberia would trace their origin to Syria but when Tarek ibn Ziyad crossed Gibraltar, it was primarily with a Berber/ Amazigh army, and even though they may have been called Arab, it is impossible to tell how 'Arab' they were. Often tribes would be 'converted' but its also hard to determine how much of these new traditions they adopted.
In greater Syria and Mesopotamia there was also a larger community of Christians than current demographics. Im not sure if Islam ever overtook Hinduism in India in terms of sheer numbers even under the Mughals. If I made a huge generalization, I would say during this time period the demographic percentage of Muslims in North Africa and the Middle East (territories under 'Islamic' rule) became the majority.
The main reason I am unable to give a super specific answer is because of Africa. Because of a dearth of written documentation outside of Ethiopia, the demographic history of Africa is much more difficult to figure. If a guy like Mansa Musa can come from seemingly out of nowhere in Mali and make it rain with gold so hard that the inflation collapses the economies of North Africa during his hajj, what is the population of his kingdom, and what percentage are they Muslim? The same problems occur in East Africa and the Swahili coast, and in South East Asia. The population of pre-Mongol central Asia would also be a good number to account for.
Has the Islamic world population ever exceeded the Christian? I'd say somewhere from maybe to probably. Best odds I think are right before the Mongols or right before the potato. The Potato made Christianity clear No 1 in terms of numbers of people.
To piggyback on to this question, during this time, were the Muslims in Europe originally people from the Middle East and North Africa, or were they ethnic Europeans whose ancestors had converted to Islam?