This one comes up so often that there's a whole section in the FAQ about it. (So often that this is the 2nd time I've used basically this exact response)

Heavily paraphrasing /u/sowser 's response (which you should go read): New World slavery was inherently racially based. Any reference you may have heard to 'white slavery' is probably referring to indentured servitude which is different because it is temporary, it arose from mutual agreements, and the person was not considered property.