It was purely linguistic: anyone who didn’t speak Greek (or Latin, for the Romans) was a barbarian.

The word didn’t originally have as negative a connotation as it does now—it was just a narrower term for “foreigner” that didn’t apply to citizens of other Greek-speaking states.

The definition of barbarian changed throughout time. As others pointed out, it originally meant someone who did not speak Greek or who was a non-Hellene. Over time it also came to mean a non-Roman.

Yet the definition was also expanded from linguistic or ethnic grounds to mean someone who was uncultured, uncivilised, rough, rude or coarse. It sometimes had religious connotations and referred to pagans or non-Christians.

It also came to mean unrestrained, which was an important distinction to the Romans who saw themselves as stoic and in control over their emotions. This meant it could apply to Romans who behaved in an unrestrained manner, such as when they got angry and couldn’t control themselves.

Most of these definitions were derogatory because they are coming from the Greeks and Romans to define foreigners, but in some cases barbarian attributes were cultivated, such as in the Roman army in Late Antiquity who took symbols of barbarian animal savagery. They did not necessarily call themselves barbarians, but some so-called barbarian attributes were seen as virtues in the military.

Sources: Oxford English Dictionary and Guy Halsall, Barbarian Migrations and the Roman West 376-568.