So I don't think there was a serious effort to convert Muslims(nor would that have been advisable!) but certainly the eastern churches produced a number of polemicists who defended Christianity against Islam. Probably the most famous are two Arabic Orthodox authors who wrote in Greek and Arabic: John of Damascus and Theodore Abu Qarrah. Interestingly, John of Damascus addressed Islam as a heresy within Christianity(claiming that he had formed his ideas out of an encounter with an Arian monk) and devotes a good deal of his attack on Islam to a rather polemical chronicle of Muhammed's career. Theodore Abu Qarrah took a very different tack in his writings and public discussions, seeking less to directly discredit Islam as a heresy then to establish the intellectual credibility of Christianity against Muslim attacks by offering a rationalistic argument(essentially, "given the following rationalist argument for the existence of god it follows that he has the following characteristics and that the true religion should have the following characteristics, christianity most exactly fulfills said characteristics) for it in the Arabic language using Arabic philosophical vocabulary. While this wasn't done to convert Muslims, it does show an effort to defend his beliefs in terms which would make sense in a Islamic political framework and indeed Muslim philosphers took him seriously enough to write refutations of his work. There are also a number of texts produced by Oriental Orthodox writers against Islam, although I am less familiar with those. A fine article on Syriac writers can be found here:

http://www.syriacstudies.com/2015/03/16/syrian-christian-writers-in-the-world-of-islam-prof-sidney-h-griffith/

Do you mean proselytize to Muslims or to others (either different Christian sets or a different faith altogether)? If I can add a follow on question, would Christians be allowed to proselytize to Muslims under Muslim rulers? I imagine not, but I really have no idea.

This isn't an answer to your question, unfortunately, but the Eastern Orthodox Church might be an interesting area of study since many of the lands where it was found came to be controlled by Muslim rulers at one time or another.

One notable factor in this is that in the earliest eras of Muslim rule, Muslims themselves do not appear to have engaged in proselytization of Christians.

Some have theorized that the early self-conception of Islam during and shortly after the Arab conquests was as sort of primus inter pares of the Abrahamic religions.

It's not until Abd Al Malik's construction of and inscriptions on the Dome of the Rock (built in 691) that we have evidence of a state-led effort to direct Islamic religious rhetoric towards Christians.

For more on this see Robert Hoyland's article New documentary texts and the early Islamic state, and his book In God's Path.