First of all, it's very important to understand that in music theory the word "mode" has existed since antiquity and has been applied to a vast number of different concepts. Unfortunately, people aren't always clear about what they mean by "mode" and the names ("Dorian," etc.) associated with it. The words have become something of a floating signifier: everybody knows that the terms exist, but nobody is convinced that they refer to one particular real musical phenomenon. (For instance, although people speak of "the ancient Greek modes," they equally often speak of Dorian etc as the "church modes" and associate them with the medieval and Renaissance musical practices of the Roman Catholic Church. For informed skepticism about the value of those terms in that context, I'd encourage you to read Harold Power's essay "Is mode real?" in the Basler Jahrbuch für historische Musikpraxis, 1992; and Cristle Collins Judd's "Modal Types and "Ut, Re, Mi" Tonalities" in JAMS 45/3 (1992): 428-467.)

As for how the names themselves worked their way into Western cultural consciousness, we do have a pretty good picture of the process by which it happened. They first show up with their contemporary meanings in an anonymous 10th-century treatise referred to as Alia musica. This document attempts to retransmit musical knowledge contained in Boethius's 6th-century De musica, but it garbled certain key aspects of Boethius's description. Boethius does discuss the 7 species of the diatonic scale (which is what jazz musicians mean when they refer to "modes" of a scale), and Boethius does use the ancient Greek ethnic names (along with one extra one, "hypermixoldyian," that nobody refers to any more). But in antiquity, the two weren't connected exactly as they are now: the ethnic names referred to certain aspects of musical style and performance, particularly how high or low your reference pitch was. (That is, imagine that "Dorian" means tuning to A440 whereas "Phrygian" means tuning to A315.) Even by Boethius's time, those ethnic names were probably purely arbitrary and without a great deal of significance. (Music terminology is, historically, full of concepts needing names that get supplied by more or less randomly borrowed sets of labels. Consider, for instance, the "Neapolitan," "German," "French," and "Italian" chromatic chords studied by college sophomores.) But Alia musica misread Boethius, applying the ethnic names to the octave species (in the wrong order!), producing the system of modal names that is more or less still familiar today.

For a long time, these ethnic terms still didn't have great popularity: most medieval and Renaissance treatises prefer instead to refer to "mode 1," "mode 7," and so on, only occasionally mentioning "By the way, mode 1 is also referred to by the ancient name of Dorian" as a show of erudition. Two 16th-century treatises helped popularize this conception of modes (Glarean's Dodecachordon and Zarlino's Istitutione harmoniche), but even these still use numerical designations alongside the ethnic names. It's not really until modes themselves have fallen out of practice in favor of major & minor keys that, in retrospect, people start using the more colorful ethnic names as the usual designator of mode. (Thus, for example, Beethoven's Op. 132 refers to "der lydischen Tonart" -- the lydian mode -- though it's clear from Beethoven's composition that his conception of what that means is worlds removed from Palestrina.)

Where do the ethnic names originally come from? We probably will never know. They are attested in what is basically the oldest extant work of music theory, Aristoxenus's Harmonic Elements, which is probably from the mid 300s BC. Although Aristoxenus documents the use of the terms, he didn't invent them: "The association of ethnic names with the octave species probably does not come from Aristoxenus himself, who criticizes their application to the tonoi by the Harmonicists." The Harmonicists, were, unfortunately, a school of musical thinkers who failed to leave behind written records & are remembered almost solely because Aristoxenus criticizes them.

That last quote is from Thomas Mathiesen, "Greek Music Theory," in the Cambridge History of Western Music Theory, edited by Thomas Christensen (CUP, 2002), 125. Mathiesen's chapter on sources from antiquity, and David E. Cohen's chapter ("Notes, scales, and modes in the earlier Middle Ages") from CHoWMuT, are the main sources I'd point you to for the content of this post.