So to start, the Christian church we're talking about here is properly known as the "Church of the East". They historically have been called Nestorians, but this gets a little contentious and complicated. Nestorius was a Bishop of Antioch who got into conflict with Cyril, the Bishop of Alexandria, over the relationship of the divine and human nature of Christ, and the proper terminology of Mary. Long and short, the Council of Ephesus in 431 sided with Cyril and declared Nestorius a heretic, but Christians east of the Roman Empire refused to attend or sanction the Council. A new bishopric was established at Ctesiphon (the Sassanian capital - it would eventually relocate to Baghdad during the Abbasid Caliphate), and after attempts at compromise the Church of the East proclaimed themselves to be their own going concern in 486.

From this point, the Church spread eastwards, not only in the Sassanid Empire but into Central Asia along the Silk Roads, most notably being transmitted by Sogdian merchants. It gained quite a bit of popularity among steppe peoples circa AD 1000, such as among the Kara-Khitai and the Kerait tribe in the Mongol confederation: Christianity for these groups seems to have been a very syncretic type of affair, with Nestorian priests more or less regarded and treated as shamans. It reached China, being allowed to operate there under the Tang in 638. Nevertheless, it like Buddhism was viewed in China at the time as a "foreign religion", and it seems to have not really caught on even as Buddhism eventually would have.

It's not really clear just why the Church of the East never caught on in Central Asia and China. While some (western) historians of Christianity have emphasized its "defective" character, or the complexity of its belief systems, these don't really provide useful answers. It has been postulated that a major barrier to its expansion was actually geopolitics: as a mostly-tolerated-but-sometimes-persecuted minority religion based in the Sassanid Empire-then Muslim Caliphate, it never really had as strong a cachet as state-supported religions.

When it finally did have a moment of state favor, temporarily arresting its decline in Inner Asia, this came through its practice by some of the Mongols (notably the Keraits mentioned above) and its toleration in the Mongol Empire. This ultimately worked against the Church in the long run in China - once the Mongol Yuan Dynasty was expelled and replaced by the Ming in 1368, the new Chinese rulers worked to eradicate a religion that had become connected with foreign rule.

At the end of the day though, in answer to the question: what happened to the Church of the East, the answer is - it is still around! After the sack of Baghdad (the Mongols spared Christians in the city by the way) the Church based itself in northern Iraq, where it has continued to this day as the Assyrian Church of the East (a separate Church broke off in 1964). The Catholic Church actually worked to unify with a branch of the Church of the East, forming the Chaldean Catholic Church (a uniate church in communion with Rome) in 1552 - this is the biggest church in Iraq (Saddam's Foreign Minister and Deputy Prime Minister Tariq Aziz was a Chaldean Catholic). The Assyrian Church of the East, Chaldean Catholic Church and Ancient Church of the East altogether have survived the vicissitudes of the 20th and 21st centuries in Iraq (including the Assyrian Genocide and the multiple wars fought in Iraq), and exist there and among the Assyrian and Iraqi Christian diasporas. A section of the Church of the East also operates in southern India, where Christianity arrived almost two millennia ago and has thrived among Keralan communities as the "Saint Thomas Christians" - those who are currently still in communion with the Church of the East are (somewhat confusingly) known as the Chaldean Syrian Church. All of these churches use a liturgy based on the Syriac language, which is descended from Aramean.