When reading pre-modern sources I’ve seen a lot of authors defining “otherness” in terms of religion, clothing, warfare, social customs, sexual customs and even hair grooming (those long haired spartans), but I don’t remember reading “they are different from us because they use these weird scales/instruments/patterns etc”. In the XIX Century certain music made by white people went from being considered a functional tool to being considered “expressive art", and it is this Nationalist context that created the concept of “folk music”, but that’s not my point; clothing is purely functional and still a tool of cultural identity and difference- tunic vs pants for instance.
So, are there any examples of people using musical characteristics to conceptualize differences in identity before European nationalism?
Now that you mention it, in the Balkans Romani were the main contributors to musical tradition. Classical music was something unheard off. Music became a part of their natural identity and thanks to them many folk songs were preserved. In fact due to their low status, the profession if a music player was treated with disdain and if a non-romani wanted to become a music player, when he died he was buried with the Romani.