This never made sense to me.
Not to discourage any further answers but you'll probably find the FAQ on racism and slavery to be interesting. You should definitely check out these posts from the faq:
Europe had normal diplomatic relations with non-white nations before turning explosively racist to justify their actions against all others. What happened to cause this shift? by /u/victoryfanfare
Race in the Roman Empire by /u/cleopatra_philopater
And most of all check out What happened to the Roman system of slavery after the fall of the Roman Empire? Was the legal basis of 18th-19th century slavery derived from Roman law, or a completely separate system? by /u/textandtrowel