By the 18th and 19th centuries Circassians were just known for their beauty. It's not uncommon even today to have regions where the women are reputed to be particularly beautiful. In modern Turkey, it's Izmir (which perhaps not coincidentally is known for the large number of Circassians and other Caucasus Muslim refugees who settled there). In contemporary America, it's arguably California (from the Beach Boy's 1965 hit "California Girls" to Kendrick Lamar's more recent song "The Recipe" whose chorus explains that people come to California for "women, weed, and weather").

The Ottoman Empire also traditionally just had slaves from certain parts of the empire who did certain things. The Kızlar Ağası, literally "the master of the girls" but more often called "the Chief Black Eunuch" in English, was usually a slave from Africa by the late Ottoman period. He was in charge of the harem. The Kapı Ağası, literally "the master of the gate" sometimes called the "Chief White Eunuch" in English, was an important position at the height of the Ottoman Empire but declined in importance after. As far as I can tell, they just become associated with slaves from different places and, from then on, they continued to be associated with slaves from those regions. Early Janissary troops were likewise taken from Christians of the Balkans, but Christians of Anatolia or Syria or anywhere else, through the devshirme system. In many ways, Circassia just became the place to get female slaves for ones harem, I'm not sure I'm ready to say that that happened for geo-political reasons. Even Western travelers were quite taken with "Circassian beauties and wrote a lo about them in the 18th and 19th centuries. European cosmetic companies might add "Circassian" to their name and P.T. Barnum even put some on tour (well, the women he put on tour weren't Circassian but he said they were).

The Ottoman spent centuries fighting the Persian Empire within the Caucasus. As a result each side forged alliances with the various rulers for varying amounts of time. To help forge these alliances, daughters/sisters would be sent to Constantinople as a means to keep favour with the Turks. Its also worth noting that unlike the other regions under Ottoman control, the Caucasus was never formally incorporated into the empire; though many regions were effectively vassals, they still nominally had independence, and thus could help explain in part why they had such a large number of consorts: it helped maintain their status.