The thing to note here is that the Pilgrims weren't the first English settlers. The first English settlement in North America that survived was in Jamestown (in Virginia today), 1607. Spanish settlement in the modern territory of the US also predated that, starting with St. Augustine in 1565.

So although servants (but not slaves) did accompany the Mayflower settlers, and they did not prohibit slavery, the 'first slave ship' (which was really just about 20 or so enslaved Africans carried by an English warship under Dutch letters of marque) went to Jamestown. These people were not deliberate imports but spoils of war, captured from the Portuguese on the high seas en route to the Spanish colonies. At the time, the Spanish and Portuguese crowns were unified, so some cross-traffic existed between the two realms, and Spain's enemies such as the Dutch Republic became Portugal's as well.