Great question! There are two main parts of the definition, but you actually got part of it yourself: From Augustus onwards, "barbarian" was used as an umbrella term for any group which was threatening the empire (So like 90% of the time it means German/Gaul). Before, that, however, it was used to refer to someone, within the empire or not, who didn't speak Latin or Greek natively and thus wasn't included in the dominant Roman culture. (Here is a nice entry which lays out the multiple meanings of the word fairly clearly).

The second part of this definition may strike you as odd. I promise I'm not going to go full philologist here, but I have to dig back a little bit. Greek has a very similar word, barbaros (in Latin text), which likewise means a non-Greek speaker and, by extension, someone who isn't Greek. After the Persian War, it came to have the very negative connotations which it carried into Latin and later English: "stupid," "outlandish," "uncultivated," or "brutal." (Wouldn't be complete without another lexicon reference).

Now comes the harder philology part. We're certain the Romans picked up this word from the Greeks. But we are uncertain as to how the Greeks got it. The traditional story which many Classics students learn as undergrads is that the word is onomatopoeic. Greeks thought people who didn't speak Greek sounded like barking dogs, going "barbarbarbar." This is an ancient etymology taken right from the Greek texts, but that doesn't mean that it's right. In fact, as Greek fluency declined in the Latin-speaking world during Late Antiquity, Roman authors started giving their own etymologies, usually relating "barbarian" with barba (beard) or rus (countryside). Isidore of Seville and Cassiodorus gave the two most famous of these Latin etymologies of the word, which are creative but clearly wrong.

But modern philologists don't buy the Greeks' etymology either. There is a very similar Sanskrit word barbara (in Latin letters), which meant something like "stuttering" or "tongue-tied," and, by extension, stupid. In a situation like this, where a very similar word exists in two different languages in the same family but there is no evidence that the word was exchanged between the languages, we call those words cognates. Cognates are "sibling words," or words which are descended from a common root and are thus similar in meaning and form. We actually have enough cognates with "barbarian" from other languages (including, arguably, the English word "babble" and the Latin word balbus meaning "inarticulate,") that we can kind of reconstruct the root which spawned all these words in the very ancient and now mostly lost language called PIE. So, our best guess at this time is that all these words come down from a PIE root meaning "stammer," or "stutter," which might have looked something like balb- or balbal-.

So, to summarize: "barbarian" really means "someone who doesn't speak intelligibly," but came to mean "some foreigner who doesn't speak my language or share my culture." Because "foreigner" was so often synonymous with "enemy," to the Greeks, it came to be a pejorative. Romans picked it up from there and ran with it.

Again, this is a really great question. Thanks!