A really wonderful book that deals with this head on is Douglas Boin's Alaric the Goth, and he basically argues that there was a shift in the late fourth century or so in which the culture of the empire became less "open", due to a variety of factors like religious factionalism and increasing military difficulties on the frontier. Particularly towards Goths, and by the year 400 (to pick one) there was something very much like modern racism towards them, including associations with slavery and brutishness. One of problems of the era is that the Goths themselves did not quite register the change, and thought were still working within the old framework in which they could smoothly advance themselves. Hence the (I believe generally accepted) idea that Alaric the Goth, the sacker of Rome, was not actually "anti Roman" but rather was a frustrated wannabe Roman who turned to military means after finding his political path blocked (itself a venerable Roman tradition).

Earlier than that is difficult to say, one aspect to bear in mind is that in the modern world a "foreigner" can pretty straightforwardly be said to be someone who comes from outside the political boundaries you are within, but to a Roman from Rome, someone coming from Greece could be thought of as a foreigner. Hence Juvenal's famous exclamation in Satire 3 that "I can't stand a Rome full of Greeks" followed by some pretty vicious bigotry. Which goes to show that there certainly were xenophobes in Rome but also, if you read against the grain, that there were plenty of people that had no problem with foreigners and "foreign influence".

As for the experience of foreigners in Rome, I do not know of a better work that the Syrian Greek Lucian's The Dependent Scholar, which is a beautifully evocative description of being a lowly Greek tutor attached to a Roman household. There are all manner of indignities he suffers, but ethnic prejudice does play into it:

Not to mention that the worst interpretation will be put upon your late dismissal; you will be credited with adultery, or poisoning, or something of that kind. Your accuser, you see, is convincing even in silence; whereas you — you are a loose-principled, unscrupulous Greek. That is the character we Greeks bear; and it serves us right; I see excellent grounds for the opinion they have of us. Greek after Greek who enters their service sets up (in default of any other practical knowledge) for wizard or poisoner, and deals in love-charms and evil spells; and these are they who talk of culture, who wear grey beards and philosophic cloaks! When these, who are accounted the best of us, stand thus exposed, when men observe their interested servility, their gross flatteries at table and elsewhere, it is not to be wondered at that we have all fallen under suspicion.

Honestly it is worth reading the entire thing, it provides a pretty unique window into Roman life. Unfortunately however there is no equivalent piece from one of the "barbarians" who came to reside in Rome.