I looked at the deeper history of labeling people "black" and white" in this earlier answer. An excerpt:
There are two basic roots of the division medieval western Christians drew between "white" and "black": geographical and moral. The first thing to keep in mind is that the two strands of 'racial' thought develop hand-in-hand, inseparable from each other. The second is that theories of race will never be neat packages and will always appear to have contradictions all over the place, because they are ideologies-in-practice and not scientific laws.
Medieval Latins had a lot of ways of mentally comprehending the Earth's geography, but by the 12th century there were two major ones. First, the idea of dividing the earth basically into 3 continents a la the T-O maps: Asia to the east, with Europe in the northwest and Africa in the southwest. This partnered neatly in Christian thought with the religious idea of contemporary humanity as the descendants of the three sons of Noah. The second, which the West gets from ancient Greece I believe by way of medieval Muslim writers (who draw on this particular view as well), is based on the ancient Greek idea of climatological zones dictating people's appearance and, eventually, behavior.
The two strands of racialized geographic thought entangle to push forward the same idea: people from the south (Africa) are black because it is warm/the sun makes it hot/they have warmer humors; people from the north (Europe and sometimes northern Asia) are white because it is cold/they have colder humors. Note that climatological determinism does not, inherently, have a moralized component. For example, Arab scholar Ibn Khaldun associates the darker skin of sub-Saharan Africans with exposure to a hotter sun, not any kind of moral difference. But throughout the Middle Ages, western Europe was developing strong associations between "white" skin and goodness, "black" skin and badness. The direct ties between Ham:curse:black skin are not entirely clear or straightforward. Indeed, central European nobles will interpret the story of the curse to tarnish their own serfs and justify their lack of freedom; the extent to which the "dark" and "darkener" imagery tied to the curse on Ham mean skin color varies throughout medieval writing.
But that shouldn't obscure a couple of basic facts. First, western Europe definitely had the theory that people from 'the south' were black; western Europe definitely had strong moral meanings attached to white and black. And those moral meanings became tied up in skin color very quickly.
The modern definition of the "white race" is a relatively recent development. For example, prior to the 18th century, East Asians were once perceived as white. This also applies to the racial classification of Native Americans, but since I specialize in Asian history, I will defer that topic to someone with more expertise.
East Asians "became yellow" as Western attitudes towards East Asia shifted in the 18th and 19th centuries. But to start with, we need to understand how East Asia was initially perceived in the West during the 16th and 17th centuries. East Asia in Western imagination was heavily influenced by the glowing accounts of early European missionaries and explorers. Circa 1549 and onwards, Francis Xavier described the Japanese as "the best people yet discovered" and "better versed in reason than other peoples." Alessamdro Valignano would designate the the Japanese and Chinese as gente bianca or white people in 1557. In his Sumario written in 1850, Valignano wrote that the "Japanese people are all white, courteous and high civilized, so much so that they surpass all the other known races of the world." Around 1512, the Portuguese apothecary Tomé Pires described the Chinese as "white like us, the greater part of them dressing in cotton cloth and silk." Matteo Ricci referred to the Chinese as albi or white, but acknowledged that the inhabitants of south China had darker skin tones. China and Japan were seen as equally "civilized" as the West. This attitude persisted into the Enlightenment. Sinophiles such as Voltaire and other philosophes would go so far as to declare that East Asia was more enlightened than the West West in governance and religion. Here is an excerpt from Voltaire's Dictionnaire philosophique:
Let us then, men merely of yesterday, descendants of the Celts who have barely cleared the forests of our savage lands—let us leave the Chinese and Indians to enjoy their lovely climate and their antiquity in peace. Above all let us stop calling the emperor of China and the soubab of Decan idolaters. We should not be fanatical about the merits of the Chinese: the constitution of their empire is in fact the best in the world, the only one founded entirely on paternal power (which doesn't prevent the mandarins from caning their children); the only one in which the governor of a province is punished when he fails to win the acclamation of the people upon leaving office; the only one that has instituted prizes for virtue, while everywhere else the laws are restricted to punishing crime; the only one that has made its conquerors adopt its laws, while we are still subject to the customs of the Burgundians, the Franks, and the Goths, who subjugated us.
Voltaire's portrayal of China was arguably more motivated by his opposition towards than the Catholic Church than a genuine understanding of Chinese culture, but it's still indicative of how positively East Asia was perceived in the West. Yet starting in the early 18th century, we begin to see early attempts at creating a scientific taxonomy for humans. In 1758, Carl Linnaeus would be one of the first to group East Asians into a separate luridus or "yellow" race. This would influence the racial taxonomy of David Hume in 1777. In Hume's "Of National Characters", he wrote that "There never was a civilized nation of any other complexion than white, nor even any individual eminent either in action or speculation."
It is no accident that the emerging idea of a "yellow" race coincides with China's declining reputation among Western commentators. But even as late as the 18th century, there was still no consensus as whether East Asians were yellow, white, or some other color. It was only in the 19th century the "yellow race" became cemented in Western imagination due to fears of the "yellow peril" motivated by East Asian immigration, anxieties over social change, and Japan's rise as an imperial power. With the yellow peril, East Asians were no longer just inferior, they were also seen as an existential threat to the West. It was this racial prejudice that would firmly link together the ideas of "yellowness", "perilousness", and East Asia. This is an excerpt from a letter written by Kaiser Wilhelm II to Czar Nicholas II:
I shall certainly do all in my power to keep Europe quiet and also guard the rear or Russia so that nobody shall hamper your action towards the Far East! For that is clearly the great task of the future for Russia to cultivate the Asian Continent and to defend Europe from the inroads of the Great Yellow race. In this you will always find me on your side ready to help you as best I can. You have well understood that call of Providence and have quickly grasped the moment; it is of immense political and historical value and much good will come of it.
By the 20th century, the idea of the yellow race had become so ingrained in the West that textbooks on anthropology and history would invariably comment on the yellowness of East Asians. Published in 1923, The Cambridge Ancient History, mentions the "The yellow skin colour of the Mongoloid man gives him protective camouflage in sandy desert and dry-grass steppe."
Sources: D. E. Mungello's The Great Encounter of China and the West, 1500-1800, Michael Keevak's Becoming Yellow: A Short History of Racial Thinking, Walter Demel and Rotem Kowner's "Early Modern European Divisions of Mankind and East Asians, 1500-1750" in Race and Racism in Modern East Asia: Western and Eastern