I'm asking in the sense that in most cultures in the world, the paler your skin was, the prettier or more respected you were (e.g. in Asia, it meant you weren't out in a field all day).
I'm not asking if europeans were inherently better (which isn't what I think), but rather when they met the native Americans or Chinese, etc., that their white skin would've given them a distinct advantage that they otherwise wouldn't have had
In Chinese sources from around the 16th century, the Europeans were not referred to by their skin colour (because honestly it's not that shockingly different of a colour than that of the average northern Asian) but rather their hair. They were referred to as "red haired" people. The same phrase was used in Japan from the 15th century as well, and has at different times referred to the Dutch, Russians, British and also generally anyone with a European appearance.
You also have to remember that pale skin isn't the same as being caucasian. You know this, of course, but I bring it up because there's this modern notion that Europeans have a skin colour that is globally different from Asians, when this isn't the case. There were surely pale Asians and tanned Europeans involved in early trade. You mention pale skin as a sign of not being in the field all say. It seems a Dutch sailor would not likely have been an example of that, since they'd be out on deck quite often. Something to consider.
For examples of the use of the red haired barbarian term in the 15th-19th century, see the following:
Lost Colony: The Untold Story of China's First Great Victory Over the West by Tonio Andrade (Chinese usage of the term)
Great Britain and China, 1833-18G0 by W. C. Costin (Chinese usage of the term)
The Private Correspondence of Isaac Titsingh: Volume 1 (1785-1811). by Isaac Titsingh; Frank Lequin (Japanese usage of the term)