I thought this was a late-medieval European theory, but I heard one of my Saudi Arabian students talk about it recently, and so I wondered where the theory originated (obviously Muslims also have the story of Noah and his sons, so could come up with the idea as easily as Europeans), and how it spread (for example, was this an indigineous Arab theory, or one spread there by say, the British Empire).
Now these are the generations of the sons of Noah, Shem, Ham, and Japheth: and unto them were sons born after the flood. [...] These are the families of the sons of Noah, after their generations, in their nations: and by these were the nations divided in the earth after the flood.
Genesis 10-11 in the Old Testament of the Bible gives the genealogy of Noah after the flood. It's a patrilineal record of descent back to quasi-mythical forebears, which was pretty common in different cultures in antiquity and right up through the Middle Ages; and as the Bible spread and was considered a significant source of historical data, it's influence can be seen in works like the Irish Lebor Gabála Érenn (popularly known as the Book of Invasions) in the 11th century, which traced the ancestors of the Irish people to:
Shem settled in Asia, Ham in Africa, Japhet in Europe. We Gaedeil are descended from Japhet.
This was "race" in the older sense of the term, as in line of descent. When somebody talked about their "race" they literally meant their father and their father's father and greatgranddad and so on, a sense that gradually assumed the connotations of ethnicity and national identity and in the 18th and 19th centuries assumed the characteristic definition of biologically/culturally distinct peoples that we usually understand it to mean today. European intellectuals and scholars in the pre-modern period, following the Biblical model, had thus been working under the general conceit that the Bible was historically true, and following that through, that all peoples were descended from Noah and his sons, and thus made up fanciful geneaologies that labeled certain groups "Hamites" and others "Shemites" or "Semites" and "Japhetites."
The term "Semitic" was revived in new use in the 18th century by A. L. Schlözer as a reference to a family of related languages - Aramaic, Hebrew, and Arabic - based on the Biblical Table of Nations. To quote Kiraz in Computational Nonlinear Morphology: With Emphasis on Semitic Languages:
It was first used by the Orientalist A. L. Schlözer in 1781 to designate the languages spoken by the Aramæans, Hebrews, Arabs, and other peoples of the Near East (Moscati et al., 1969, Sect. 1.2). Before Schlözer, these languages and dialects were known as Oriental languages.
From the idea of a Semitic language family eventually emerged the idea of a Semitic people or race. In the 18th and 19th centuries, the Bible-as-history was being challenged on multiple fronts, and "race" began to take on a new conception. Under many different racial categorizations, broad divisions of humanity were defined to be Caucasoid, Mongoloid, or Negroid. By most designations the majority of European peoples were labeled "Caucasian" or "Caucasoid"--and therein creates a problem, because there were relatively "light-skinned" people that outwardly resembled Europeans living in places like Africa, like the Berbers and Egyptians of North Africa. Just as Arabic, Aramaic, and Hebrew has been classified as "Semitic" languages, so were Berber, Cushitic and Egyptian classified as "Hamitic" languages; and just as Jews and Arabs became classified as "Semites" or Semitic peoples, so were various North African peoples classified as "Hamitic."
This is a relatively important distinction because from the 19th century on, "Hamites" are explicitly Caucasian peoples living in Africa, but prior to that the Biblical basis for the name had been used as a supposed origin for black Africans and/or a justification for slavery - "Cursed be Canaan! The lowest of slaves will he be to his brothers." in Genesis 9: 20-27. Following the new designation of Hamites as basically white Europeans, this branch of scientific racism gave way to an anthropological theory called the "Hamitic Hypothesis" - essentially an argument or assumption that much of the technological and cultural development which was being revealed by studies in Africa had been the result of "Hamites"; thus reinforcing the idea that "Negroids" were incapable of independent cultural development, which in turn supported Colonial expectations of "exporting" civilization to the "savages." Edith R. Sanders quotes in her The Hamitic Hypothesis; its origin and functions in time perspective [PDF] one passage:
Apart from relatively late Semitic influence... the civilizations of Africa are the civilizations of the Hamites, its history the record of these peoples and of their interaction with the two other African stocks, the Negro and the Bushman, whether this influence was exerted by highly civilized Egyptians or by such wider pastoralists as are represented at the present day by the Beja and Somali ...The incoming Hamites were pastoral 'Europeans'-arriving wave after wave-better armed as well as quicker witted than the dark agricultural Negroes.
Notice that the idea of "Japhetite" has largely disappeared; Japhetic was used by some linguists to refer to the Indo-European language family, and was adopted by Soviet linguist Nikolai Marr for his "Japhetic theory," which argued that Indo-European languages were formed from the synthesis of more ancient European languages (Japhetic languages) with others (i.e. Semitic languages, Basque, etc.). Marr politicized the theory by marrying it with Marxism, which didn't go down so well with critics of Marx (including Stalin), for more on which see The fate of Nikolai Marr’s linguistic theories: The case of linguistics in the political context. Since Marr, the term has generally fallen out of favor, just as "Hamitic" as it was conceived has likewise fallen out among anthropologists and linguists.
I can't tell you explicitly how this worked in an Islamic context, but that should give you some idea of how the idea developed and spread.