The simplest answers are proximity, precedent, and ocean currents. A good current system aided the circular trade flows around the Atlantic, and the major settlements were all on the Atlantic side anyhow. The viability of African plantation slavery arose from sugar islands off the African coast, a system that transplanted to the Americas in the mid 1600s. Convenience and inertia were the driving forces, because any of those other possible sources didn't have existing trade relations or would have made the voyage so long as to be totally unprofitable. See John Thornton, Africa & Africans in the Making of the Atlantic World (1998) or his more recent cultural history of the Atlantic. Phil Curtin gets more into economic specifics in Rise and Fall of the Plantation Complex.

(edit: keep in mind that labor recruitment and enslavement in other places and times followed this argument too, whether in Dutch VOC Java, the US West of the transcontinental railroad, the importation of Indian labor to British East and South Africa closer to 1900, etc.)