I can tell you specifically about Muscovy, and later the Russian Empire, struggles against nomadic people. First of all, you have to understand that it wasn't always possible for a pre-modern State to always garisson its border defenses, epsecially if it had a great border such as Russia's. The nomadic people exploited that to the very maximum, striking defenseless villages and cities when resources were directed elsewhere, taking captives and sacking those places in the process, which at the same time slowed down the Russian economy and boosted theirs (Russia frequently ransomed those peasants taken as slaves, see the Ulozhenie of 1649 take on the matter). The Crimean Khanate, for instance, managed to sack and burn even Moscow in 1571 due to the war effort against Poland in the Livonian War, taking 60.000 slaves in that raid alone. Poland and Russia often payed the nomadic peoples to attack one or the other, which further boosted those tribe's tresaury.

Taking the war to the nomads was nighly impossible until the second half of the XVIII century, as Peter I disastrous expedition to Khiva in 1717 proves. The supply lines were too thin, impossible to be defended correctly and the Russians didn't dominate steppe warfare well enough to compete with the nomads in that regard. You have to take into account that in most nomadic societies the tribes ARE the army. There are no farmers, no settlers, no differentiation between soldiers and male civilians, basically. So they had sometimes HUGE numbers advantage over the Russians and I would say even a logistical advantage against any settler nation that had to raise and mantain a regular army.

I also wouldn't underestimate the power of horse archers. In fact, the Russians mantained some nomadic people, such as the Kalmyk, as special troops in their army well into the XIX century. There are even reports of Kalmyk horse archers performing excedenly well against the French army in the 1812 invasion of Russia.

The solution found by the Russians was the gradual colonization and fortification of the steppe lands, creating an advanced defense line against the nomads, which was a slow and very expensive process. That and the modernization of the army eventually tipped the scales in Russia's favour, but it wasn't easy at all, and we are talking about a somewhat modern State against fractured nomadic factions, mostly former parts of the Golden Horde. I can't really imagine how scary it was to defend against an unified Mongol Empire at its full might.

Well, I think where your assumption is incorrect is in thinking of the Persian peoples of this time period as "settled" in contrast with Mongol horse archers.

It's true that the vast majority of the populations of these territories, stretching west from approximately modern Afghanistan to the Med and beyond, were settled and had established large, prosperous cities.

But going back to at least the 9th century and the "Anarchy at Samarra", formalized in the 10th century with the Samanids and continued by the Buyids, Seljuks, Ghaznavids, Khwarezmians and a host of other smaller dynasties, the actual military rulership of the Persianate (and for that matter, most of the Arabic) world was in the hands of Turkic steppe peoples from the east. These peoples were very similar to the Mongols in most respects, and when you consider the relative ease with which they defeated and displaced one-another it's less surprising that they could be defeated and displaced by the Mongols.

The more interesting question, then, is not how these groups were defeated but rather why the Mongol conquests were so monumentally destructive in comparison. There are different reasons for this. One is scale, the Mongols obviously amassed a much larger and more powerful empire than virtually any steppe power before them. Furthermore they were savvy enough to employ settled peoples with expertise in attacking cities and had plenty of experience in dealing with cities from their conquest of China.

But in my view, and that of many historians, Islam was the really big differentiator. All of the other steppe dynasties I mentioned were already Muslims at the time they conquered. So even if there was a nomadic/settled Turkic/Persian divide, the Islamic elites of Persia were able to mediate and facilitate the new rulers, so that even if the dynasty changed, law, order, taxation, treatment of the conquered peoples, basically all of the social aspects of actually living under a state were virtually unchanged despite the succession of dynasties who exercised control from the 9th to 13th centuries. Hodgson terms this the Amir-Ayan system, which you could translate as something like "Military Commander-Local (Islamic) elites and notables" system.

The Mongols break that tradition. Once the Amirs were defeated in battle the Ayan did not understand they were dealing with something new, and even if they had it would not have helped because the Mongols didn't care about respecting the societal norms of Islamdom. As such, not only were the conquests themselves uniquely brutal in the history of the region, but their post-conquest rule was similarly disruptive. The negative impact of their rule lessened significantly after they converted to Islam and started working with local advisors/viziers like Rashid ad-Din Hamadani at the end of the 13th century.

edit: typo

Lacertoss makes a good point about just how long it took Russia to overcome the various steppe peoples they had to contend with and the continued military utility of horse archers. Chinese history is punctuated with periods of nomadic military ascendancy, and gradual colonization of the steppe with fortified settlements was one of the only effective ways to deal with them there as well.

The military effectiveness of nomads vis-a-vis 'pre-modern' settled peoples cannot be over-stated, and is made obvious by how often the ruling dynasties in settled areas surrounding the Eurasian steppes where in part or entirely descended from nomadic conquerors. In China alone the most notable examples include the Han, Tang, Song, and Ming intermarrying with nomadic elites at times, and the Jin, Yuan (Mongol), and Qing dynasties established by nomads, just to name a few.

From the first domestication of livestock in pre-history pastoralist groups broke off from agrarian societies, and on the whole lived a far better life than farmers through most of pre-modernity. The mobility this lifestyle gave them - especially with the domestication of horses between 3500-2000BCE in what now is the Ukraine - distinct offensive advantages over settled populations, in that they could simply ride in, take what they wanted, and then disappear. Although these advantages were ameliorated over time by the development of fortifications and state structures which provided better defences for settles populations (including the employment of their own cavalry forces, often recruited from pastoralist groups), border zones remained subject to nomad raids and periodic invasions. Over time technological advances (some indigenous, and some simply taken from agrarians) increased the military effectiveness of nomads including more powerful composite bows and (successively) the chariot and stirrups. This was compounded by developments in horse breeding, which - contrary to the development of large, powerful warhorses for shock cavalry in settled societies - often focused on speed and stamina.

The Eurocentrism of Western historical traditions has placed an emphasis on the dominance of shock cavalry like knights in the Middle Ages, but in the majority of Eurasia during this period it was horse archer that dominated wherever possible. Western Europe was an exception to the norm partly due to its geography, as it was relatively shielded from the steppes by rivers, mountains, and forests in which horse archers were less effective, and there was limited pasturage for the mass of horses which accompanied a nomad army. In any place exposed to steppes, desserts, or other semi-arid zones, nomadic cavalry played a prominent - if not always dominant - military role in pre-modern times.

Another exception to the norm was the eclipse of heavy cavalry by gun-armed infantry in the so-called Military Revolution in early modern Europe. Although a similar revolution in tactics stressing the combined use of early cannons and massed, drilled infantry had occurred in late medieval China it had not been effective enough against nomads for a variety of reasons (including supply issues) and had not sustained itself. Again, European geography played a role in sustaining the Military Revolution in its early days, with similar developments flourishing only in Sengoku Era Japan, which was also sheltered from the steppes.

Even with the development of effective gunpowder weapons and drill the horse archer remained a superior offensive soldier, who could ride in, fire multiple arrows, and then withdraw all during the time it took to reload and fire a matchlock gun. Indeed, the development of gunpowder even helped nomadic armies at times, with the combination of massed horse archers and artillery, as demonstrated by the Mughals at the First Battle of Panipat allowing them to fight even more effectively against settled populations and leading to the establishment of early modern Gunpowder Empires in the Islamic World.

Western-style infantry-centric armies were only able to engage with and defeat numerically superior cavalry-centric Asian armies on a regular basis after subsequent technological and tactical advances and only did so on a large scale from the 18th Century onward. The developments included flintlock guns, bayonets, advanced in artillery, and combined armed tactics which incorporated units of light cavalry (often - as with the armies of the East India Company - recruited locally).

If you're interested in the military effectiveness of steppe cavalry vis-a-vis infantry or the Military Revolution in an Asian context more broadly good places to start reading are the classic The Pursuit of Power by William H. MacNeill, The Asian Military Revolution by Peter Lorge, and any of the anthology texts relating to the Military Revolution debate among historians like Geoffrey Parker and Clifford J. Rogers in the 1990s. Jeremy Black also has some good stuff on the delayed development and/or effectiveness of the Military Revolution as exported to Asia by the British.