I think there are really two almost separate sets of questions here:
Did European slave traders tap into pre-existing Arab slaving routes to feed the Transatlantic slave trade in 16th-18th centuries? If so, to what extent and with what effect on the overall transatlantic trade?
How were Muslims defined under European international law? Were they liable to enslavement by Europeans? What about sub-Saharan African Muslims?
I honestly don't have a very satisfactory answer to the first issue. My sense is that Muslim slave traders did have some involvement in at least the Malagasy trade, and that European slave traders, particularly the Dutch and French, also traded extensively in East Africa and Madagascar starting in the late 16th century. You might want to check the transatlantic slave trade database (slavevoyages.org) and see when and to what extent European merchants purchased slaves in this area, and/or if they had any specific contact with Muslim merchants.
The second set of questions are a bit more complex and difficult to answer completely. First off, I think we need to recognize that "race" meant something very different in the early modern world - it was more closely associated with nationality (or in some cases religion - European Jews were often described as a separate "race" in this sense) than with skin color. (See, for example, Daviken Studniki-Gizbert, A Nation on the Ocean Sea) This, however, was a crucial element in European national and international law, and largely set the terms upon which slaves were held. (See Richard Tuck, The Rights of War and Peace)
In general, European international law allowed for the enslavement of "strangers," typically defined as non-Christians throughout much of the early modern period. Furthermore, the theoretical basis for slavery was the "just war" theory, which allowed the victorious party in a "just" war (and all war against non-Christians was defined as just) to enslave captives rather than execute them. Under this definition, any non-Christians, including Muslim Africans, were liable to enslavement. There is some very good evidence that relatively large numbers of slaves imported to the British mainland colonies (the future US) were first enslaved during a series of jihads in the Futa Jalon region of what was, at the time, called Guinea. Many non-Muslim Africans who refused to convert to Islam were enslaved by the Futa Jalon and sold to European merchants on the coast. (Sean Kelly has a forthcoming book dealing with this topic - blanking on the title at the moment.) In this sense, then, Islam and its expansion into Africa had a great deal to do with the transatlantic slave trade by exacerbating and expanding intra-African wars that resulted in enslavement.
To most European merchants, however, the origin of slaves was a decidedly minor concern. Kongo, for example, had officially converted to Christianity in the late 15th century, but Europeans continued to purchase Kongolese slaves, many of whom were Christians, throughout the colonial period. (See John Thornton, Africa and Africans in the Making of the Atlantic World) To overcome what could have been a thorny legal issue, first Virginia in 1667, then all English slave colonies shortly thereafter, passed laws stating that Christian baptism would not change a slave's condition. This, it seems to me, is a crucial turning point in addressing your question. Before this point, religion appears to have been the crucial deciding factor in whether or not a person was liable to enslavement -- following this statute, it appears that the focus shifted increasingly to a more recognizably modern definition of race as skin color.