So it is always tricky to work with these questions that span such vasts amounts of time and space, but speaking with respect to the empires I am familiar with, I don't think you are entirely mistaken.

There are two patterns arguably at work here, one of which your premise explicitly denies: tribes were not necessarily "harmless". Many possessed what is called the nomadic advantage, namely, a lifestyle that resulted in the average man (and in many cases women as well) possessing a skillset and attributes that made them potentially excellent soldiers. Even in cases where the tribes were not nomadic, this still applies to some extent. Exactly which attributes these were varies with the group and author, but a couple of commonly mentioned one:

1. Lack of permanent settlements, leading to various survival skills and self-sufficiency.

2. A "martial lifestyle" potentially including both traditional martial arts, horsemanship, and value placed on skills with weapons such as the bow, which may also be important to survival (e.g. for hunting).

3. Experience with raiding. Many such groups were to some extent dependent on settled populations for luxury goods and surplus food. These could be attained through trading, but also by plundering settlements or outposts if trade was not profitable enough. Often, the merchants and the raiders were one and the same (in some cases, as with the vikings along the Volga river, they could be intimately tied together - raid one settlement for slaves, then sell the slaves elsewhere).

4. A high value placed on personal honor, oaths and covenants, and/or a strong notion of masculinity tied to proficiency in the above three categories, often defined in opposition to the "meek" and effeminate settled populations. (A personal observation is that many such groups - Arabs, Germanic peoples, etc, also seem to have possessed a strong sense of legalism, which I suspect to be tied to the idea of personal honor being important, but perhaps also to the kind of social structures needed to maintain some manner of social control against vast and inhospitable lands.)

Compared to an army mostly composed of levies from settled populations, nomads could be absolutely devastating. They were not invincible - when faced with a comparably sized group of martial elites or professional soldiers, such as with the Mongols against the Mamluks at Ain Jalut, and perhaps with the Umayyad armies in Gujarat, they could absolutely be defeated (even crushingly). But that requires a strong opponent with a particular set of social institutions, and of course, like anyone else, nomads prey on their enemies when weakened. (It has to be asked though, that if the Khwarezm-Shah had taken the Mongol threat seriously and marshalled the full strength of his Mamluk forces, would he not have stood a good chance against Chinggis Khaan?)

The second pattern has to do with the effect of empires on surrounding populations. Consider the quintessential empire of antiquity, that dominated its surroundings to extents unheard of. I am of course talking about the Achaemenid Empire (what, you thought I meant Rome?). Following and preceding their fall, what do we see in the world around them? We have Porus around the Kush, and the Maurya Dynasty for the first time uniting almost all of the Indian subcontinent (and Sanskrit also acquiring a writing system by way of the Achaemenids). We have the Scythians and Xiongnu of the Steppes. We have Macedonia in Greece (once an Achaemenid client state). Carthaginians in Northern Africa, Kushan in East Africa. Why do we see the emergence of such powerful states around the fringes of the Achaemenids? Well, the nature of an empire seems to leak outside its borders - not only does a stable empire mean profitable trade routes from aboard, but also the spread of imperial ideology, structures of social hierarchy, and the opportunity for local elites to use their powerful neighbour to leverage control over their local territory. Though they do not fit into the pattern of being nomads, such a spread of ideas, tradition, wealth and so on is notoriously obvious in Classical Greece, in spite of their successful repelling of Persian armies. (Perhaps the Perso-Elamite kingdom of Anshan, neighbour to the great Neo-Assyrian realms, that would found what was to become the Achaemenid Empire, would be another apt example.)

Tying these two together, an empire not only means that the nomads could get rich from trade and raiding more prosperous trade outposts than ever, but also that they have a powerful neighbour willing to hire them as mercenaries, resulting in experience with military organization and potentially further spread of ideology. I don't know whether one should count the Achaemenid use of Greek mercenaries to this, but it was certainly the case that the Sasanians and the Romans both used Arabic (and Turkic) mercenaries; and in more recent times, the Qajar and Afsharid tribes were both used by the Safavids as an important source of military power; both would later rule Iran. And the use of "Romanized" Germanic chieftains (such as Flavius Odoacer himself) as military commanders by Rome I doubt I need to speak of at length.

By nature, any search for broad patterns will be speculative in nature, and studying nomadic and "tribal" groups comes with the inherent problem of poor records. Moreover, specialists in any area will always be able to point to ways in which their time and place of expertise deviates from the idealized pattern. Nevertheless, the social and economic dynamics at work certainly seem to be valid for a vast swathe of the past 2500 years!