Did the British and French take a more tolerant approach of Muslims in their African colonies compared to "pagans"? Or did they try to convert them equally and simply had less success?

Yes, British and French colonial administrations were content not attempt conversion of Muslim communities in their African colonies.

There were a handful of interrelated reasons for this attitude:

1.Notions of Hierarchy of Civilizations- in European anthropological thought popular in the later 19th century, there was a prevalent idea of categorizing peoples into grades from "most civilized" to "less civilized" to "primitive".

It was an article of faith (pardon the pun) among Europeans (and Anglo settler societies in North America and Australia/New Zealand/South Africa) that European culture represented the pinnacle or vanguard of civilization, and that Christianity was an integral aspect of what made Europeans civilized, and therefore the spread of Christianity through missionary activity was part of the British and French (and German and Belgian etc) "civilizing mission".

But as a corollary, British and French anthropologists and colonial administrators considered Islam and Islamic societies to be somewhat "less civilized" than Christianity, but nevertheless much more civilized than "pagan" peoples.

This ties in closely to the second point:

2.concessions to Indirect Rule- Colonial administrations generally ran on shoestring budgets and (after the initial military conquest), with minimal amounts of European administrative manpower.

In order to make this system work, the British and French administrations relied on "native chiefs" to serve as local administrators and the lowest level of the colonial bureaucracy.

For example, after the conquest of the Sokoto caliphate and the formation of the colony of Northern Nigeria in 1900, the British then recognized the legitimacy of the Emir of Kano and the Emir of Zazzau (Zaria) in exchange for their submission and loyalty.

As Muslim princes, these Emirs insisted on certain conditions for the protection of Islam in their territory. Terms like the recognition of Sharia courts as the law of the land, respect for madrassahs as the educational system, and keeping out Christian missionaries.

3.the fear of Islamist (or Mahdist) revolt I am more informed about this as a British concern. The British experience suppressing the Mahdist state (from 1881-1899) in Anglo-Egyptian Sudan strongly informed their later policies. British administrators were very careful to cultivate links with "moderate" religious leaders in hopes of marginializing any voices that would try to revive the Mahdist message. Additionally, British administrations were very careful to avoid any provocative actions, and blocked the efforts of missionaries to build a mission school in Darfur, citing the Islamic character of the region. (of course, the British considered southern Sudan to be a non-Muslim region and were active in promoting missionary activity in that region).

So, broadly speaking, British and French colonial administrations were careful not to allow missionary activity in areas that they considered to have an established Islamic character.

The end result of these policies was that in these Islamic regions of Africa, there was something of a religious revival during the Colonial era, with a proliferation of religious orders in Senegal and northern Mali, for example. Additionally, "pagan" communities which lived in those areas designated as traditionally Islamic by colonial authorities frequently converted to Islam in this period. There were numerous reasons for this: the appeal of Islam as a trans-national religious community, the vitality of these new religious orders, or because Islam was discernibly not the religion of the European administration and thus could viewed as a form of resistance.