Before I get to the directly relevant history, I should probably make a basic point about the stereotypes we develop around skin colours. Black people do not actually have black skin. We should not assume that 'black' has always seemed like a reasonable term when 'yellow' has not. In the USA, the use of 'black' has been challenged a number times, most recently with the now reversed ascendancy of the term 'African American'. The fact that they are referred to as 'black' is the result of a widespread crude approximation by Europeans that began in the Middle Ages. Such crude approximations result partly from a comparison to ones own skin tone and partly from broader conceptions of ethnic groups.

While dividing the human race into four or five 'colours' is largely a development of the modern era, there are earlier precedents. One well-known example is the medieval rabbinical division of mankind into the races of the three sons of Noah. This later came to be applied to modern conceptions of races (e.g. black people are the sons of Ham), but this application is, in part, anachronistic.

Applying the term 'yellow' to the skin colour of an ethnic group can certainly be offensive, partly because we often associate the colour yellow with ill-health. That said, some skin is certainly yellower than others. My father and myself both have almost entirely white British and white Irish ancestry, but his skin tone is certainly yellower than my own. Labels for skin-tone exist entirely in context; well into the early modern period, European literary writers typically used 'white skin' to differentiate individual Europeans with particularly pale skin. To many Northern Europeans, the skin-tone of an East Asian person might seem relatively yellow compared to their own. However, they might say the same for someone from Western Asia or Southern Europe. As you point out, an entirely neutral observer would certainly not label East Asian skin yellow. Early European travellers to East Asia, most famously Marco Polo, often wrote that people there had white skin. The adjective 'yellow' was not used. That said, the application of 'yellow' to skin colour was not entirely a modern, European imperial contrivance. The seventeenth-century German historian Georgius Hornius labelled the skin-colour of people from the Arab world and South Asia as yellow.

The classification of East Asian people as 'yellow' is a result of classification itself. It was initially popularised not in isolation, but as a 'scientific' label for a phenotype that fitted with existing terms such as 'black' and 'white'. Carl Linnaeus's 1758 edition of an earlier work, Systema naturae, was arguably a key publication in the popularisation of using the word 'yellow' to classify Asian people as yellow, but the application of it specifically to East Asian people was a gradual development that only became a dominant terminology in the nineteenth century. It was also only in the mid-nineteenth century that scientific racism began to typically consider East Asian people a distinct and single race, one often called 'Mongoloid'. Michael Keevak's Becoming Yellow: A Short History of Racial Thinking describes these developments and is the long answer to your question. While I would argue that it possibly overemphasises particular Europeans' descriptions of particular East Asians as 'white', and sometimes takes a tone that implies conspiracy, it is a fascinating read.