It's a sensitive topic, I know. But I can't think of any enslaved group in which the exploited were lighter in complexion than the group enslaving them. American Indians, Africans, Chinese... These all seem to follow a troubling pattern.
Racial slavery is a comparatively recent development that did not exist in antiquity or to my knowledge in the Middle Ages. In the ancient world slaves were taken from wherever they happened to be either captured and sold by slavers or taken as captives in war. A slave at Rome was at least as likely to be a fair-skinned Celt or German as he was likely to be a darker-skinned Nubian or Parthian. Not that the stereotype did not necessarily exist--Cicero poked fun at Piso's supposedly servile swarthy complexion--but race and skin color in reality had little to do with slavery in antiquity--Horace's father, an Italian probably enslaved during the Social War, would have been indistinguishable from his Roman masters. And even in the case of jokes like Cicero's the situation is more complicated than simply skin color--Cicero accused Piso of being descended from Syrian slaves, by which Cicero means Greek Syrians, who would not really have looked too different at all
You may be interested in a couple of earlier comments on the Black Sea slave trade where, among other things, a slave could be valued for being "exotic", and "exotic" could mean pale hair/eyes/skin.
When did Scandinavians (particularly the women) begin to be stereotyped for their beauty? In history were there other countries in Europe that were regarded by the rest of Europe to have the most beautiful women? - /u/spin0 gives a general introduction re Scandinavian slaves
Where did Ruxelana (aka Hurrem Sultan) come from? - /u/mikedash (author of one of the articles linked by /u/spin0 above) focusses on a single enslaved woman, Ruxelana
edit: this thread may also be of interest