Was it just because they were overconfident and overlooked the threat from these invading nomads?
They were less educated, less advanced in terms of technology, and yet they were able to cause empires to crumble, empires with thousands of years of military wisdom. The Mongols were even able to invade and defeat Poland on 2 separate occasions.
What exactly made them such a terrifying force?
Military power is not proportional to “civilization” (i.e. the quality of stratified societies that bother to produce literary sources for historians to analyse). Pashtun tribal warriors in the mountains of Afghanistan are nowhere near the “sophistication” of the coalition they faced, or used to face; still, they seem to be pretty successful. Military power is about the adaptation of a given society to war—and nomadic people were very good at it. The steppe world of central Eurasia, from Mongolia to the Ukrainian Alföld, is an endless band of grass, which is very suited to pastoralism, and quite improper at anything else. As a result, mere alimentary diversity requires either trade with, or predation on neighbouring sedentary societies. In both cases, nomads can rely on their main assets: horses. Given the sheer immensity of their territories, the ease to feed them, and the need to move their herds to find pasture, nomads of the temperate zone naturally used horses.
Every single member of a tribe was a trained horseman; and natural tension between tribes, or occasional raids, made the use of the bow a quite natural addition to this first skill. And this combination was, as Denis Sinor puts it, the pre-gunpowder ultima ratio regum: groupings of nomadic groups could marshal dozens (hundreds) of thousands of horses, mounted by skilled horsemen. Raiding was a natural extension of their normal mode of living. In contrast, the settled societies you mention simply could not put on the field too high a percentage of their population on the battlefield; and even the limited quantity of soldiers they were able to enlist could not compete with the average letality of mounted archers. Once you integrate these data, in fact, your question can even be reverted: why weren't nomads more successful? Mainly because they relied on their link with a specific ecological niche. Outside of the steppe world, the number of horses required (at least 4/5 by warrior, Marco Polo mentions a number as high as eighteen) could simply not be fed. This transitional phase between nomadism and de facto transformation in a settled power was the hardest part—the Huns' experience was an utter failure, while Mongols and Turks (esp. the Ottomans) arguably were the most successful (unless you want to include the Arab conquest as well).
Sources (these are classic articles on the theme):
R. P. Lindner, “What Was a Nomadic Tribe?,” Comparative Studies in Society and History 24 (1982)
D. Sinor, “Horse and Pasture in Inner Asian History,” Oriens Extremus 19 (1972)
I can't answer your question in detail, but the book Why The West Rules – For Now by Ian Morris attempts to deal with this issue: he tries to graph social development over 15,000 years and compares the traditionally-labelled "East" with "West" to discuss where things changed over. He talks a fair amount about nomadic tribes and the way they were integrated and used as pawns by larger civilisations, eventually crushing Rome and China. You'd probably find it fascinating – I think I came across the book via this very subreddit and although I'm only halfway through it's absolutely fascinating and very well-researched.
hi! you may be interested in responses to these related questions
How did the numerious invasions of steppe peoples happen/why/are they as similar as they seem?
How did the Mongols overpower a lot of great civilizations, with just the horseback archer?
Was there ever a Western force that decisively defeated/pushed back a Mongol horde/army?