Like you say, it was applied to many people. In the context you're using it in, the term more or less meant "non-Romanized/Hellenized person." Which means everybody living outside the Empire's boarders, and often a hefty chunk of the people living within those boarders.

As such, it's nearly impossible to provide anything like a concrete answer to your question. I can say that, on the whole, we're not talking about Conan here. These were complex societies with their own culture, traditions, political structure, towns, trading relationships, and all the other trappings of what we might call ancient civilization.

You could say that a more or less consistent trait of Barbarians was that they had significantly smaller political units than the Romans did. (Which is hardly saying anything, given the eventual size of the Empire). Many Gallic, Celtic, and Germanic peoples (for example) were organized into tribes, which came into conflict with one another. So while Gaul was a massive territory, the Gauls were not really a single people. Against these groups the Romans were effective primarily because they were good at presenting a united front, while encouraging their neighbors to remain divided.

But even then, that isn't really true for all barbarian peoples. The Parthians, and later the Sassanids were fairly massive political groups. They often proved a match for Rome militarily, were wealthy, and usually pretty good at presenting a united front when they weren't in the middle of a succession crisis. But to the Romans, they were still "Barbarians."