There is a significant difference in implied meaning between "mistress" and "concubine" that I expect covers at least some of the examples you're thinking about. Without knowing more about the particular ones you have in mind it's not possible to know if it covers all of them, but once the association has been set up that "exotic = concubine" it's not surprising if that's how the distinction continues to be made, even if it's less precise.

The word concubine is Latin in origin and the word concubina described a cohabiting relationship without marriage. The important thing is that concubinage came to be an officially recognised practice, with notice having to be given to the authorities of the relationship. It was used for relationships where marriage was not permitted or not desirable - for example a senator in relationship with a freedwoman, or wanting to prevent the children from the relationship having a right to inherit. (I only mention Rome first because it's the source of our English word and this question is primarily about language, not because the practice in Rome is somehow most important or influenced any others.)

It is this official status that distinguishes the meaning of concubine from that of mistress which, still today, has the connotation of an illicit relationship, even if it may be an open secret in some cases.

So we see for example that China only outlawed the practice in 1949, Japan in the Meiji period, and there are plenty of examples in the Arab world and elsewhere of harems of women who have some status akin to, but not the same as, wife. The thing they have in common is some level of recognition.

This is distinct from prominent examples of mistresses in the West like Henry VIII who (to put it mildly) famously had to divorce each wife in turn in order to officially marry the next one, even if they were engaging in some extracurricular activities before it was all official.

That is not to say by any means that all these practices which are described as concubinage are the same by any means; they weren't (but I can't really comment on the ways in which they are different). However, to Western Christian eyes there is a clear unifying theme that meant they are described the same way and in contrast to the prominent Western practice. That's not quite the same as saying "we are superior and therefore must be different" but that the lens through which we view other cultures shapes which differences are seen as important.

So this is a slightly different issue than just a linguistic difference. I don't feel like it is able to be adequately addressed without setting some parameters. You state that mistresses are "extra-marital affairs". This implies no legal binding to their status. So for the sake of this we'll agree that any non legally allowed partners are mistresses. This however leaves a large amount of grey area which I'll try to explain a bit more.

Firstly, the concept of a powerful man having multiple legal sexual partners is not unique to non-Western cultures. This is especially prominent in Europe following the fall of Rome. Now to simplify we'll say everyone in former Roman areas were Christian (this is far from true). But the issue of Christianity gets complicated because in the early medieval period there were multiple forms of Christianity each allowing different things. In addition marriage for centuries in the medieval world was not considered an official sacrament and was thereby not considered part of the church's domain but rather a private matter.

So if marriage was private, and the various aspects of Christendom had different rules about monogamy, then how do we know this? We know this because of inheritance laws! In the Lombard laws, there is a provision for the children of wives, the children of concubines, and bastard children given status. This means that if bastard children were given a different claim than those of a concubine, then the children of a concubine must be different somehow than those of a woman who bore the man a bastard. This is further exemplified in the laws by stating how a woman is to be given the ability to deal with her inheritance. A wife as well as a concubine are meant to allow their man to oversee their inheritance and dowries if applicable even if he isn't allowed to always dispose of it as he wishes. Whereas the woman who bears his bastard children gives no part of her inheritance to him and is exempt from his over sight. This implies a legal status for a concubine that gives her a household rank. The same can not be said for a woman who with no legal claim to the man still has sex with him i.e. a mistress. This is seen at countless points in the Merovingian dynasty especially.

As to non-Western cultures the same rule can be applied. If the woman he is sleeping with holds no legal status in his household and is not being paid to sleep with him (prostitute) then she was a mistress. This is the ultimate difference and the reason why the terminology is different in Western society is the general acceptance of monogamy, this acceptance meant there was only one legal position available per man and thereby anyone else was in fact a mistress by definition. This monogamy was not so generally accepted in Eastern societies where women could still hold legal status in a household outside of marriage. This is once again seen by Chinese provisions on inheritance laws regarding the rank of the woman in the household and making a separate provision for bastard children.

TLDR The west was legally monogamous and the East wasn't.