Linguist here.

Barbaros probably didn't refer to a particular people group. It has been reconstructed from the Proto-Indo-European root *barbar- "echoic of unintelligible speech of foreigners (cf. Sanskrit barbara- "stammering," also "non-Aryan," Latin balbus "stammering," Czech blblati "to stammer")." Considering that *barbar- has derivatives in multiple branches of PIE, it's unlikely that it was a borrowing or that it was an independent development multiple times.

That said, barbaroi, while technically meaning "all those who aren't Greek," did refer to Medes and Persians quite frequently (incidentally without pejoration until after the Persian wars). I think a relatable example would be "American" referring to a citizen of the United States even though anyone on two continents is technically "American," although perhaps the term "American" is more closely associated with the United States than barbaroi was with Persians.