This may be an ignorant post and I'm not condoning slavery at all... but how did native Americans avoid slavery? Were they just too good of fighters to try to keep under control?
More can always be said, but this answer by /u/anthropology_nerd explains that many Native Americans did not avoid slavery.