To my knowledge, the Muslim forces inflicted a severe defeat on the Chinese at the battle of Talas in the year 751. Furthermore, 4 years later, China plunged into a state of civil war during the An Lushan rebellion. So why didn't the Muslims take advantage of this period of political turmoil to invade China?
Off the top of my head (I'll still post sources), I can recall the Battle of talas was in Central Asia, and by winning it the Abbasid Caliphate got a sweet piece of territory on the Silk Road. This was Sogdiana, and if the Abbasids had THAT, there wasn't much impetus to spread further east. Central Asia was technically more desirable to an empire in the geographic location that the Caliphate was than China proper, and in Central Asia a major entity at the time was Tibet (and the Umayyads prior to that). Any incursion into China proper would bring the Abbasids into conflict with the Tibetan Empire, who was taking advantage of An Lushan to grab what territory they could. Instead, the Abbasids used this period to form an ALLIANCE with Tang China, rather than invade it. Harun Al-Rashid sent a group of mercenaries to help the Tang put down the rebellion, and while I don't have my chinese keyboard on this computer they became known as the Black-Robed Arabs to the Chinese. Afterward, Rashid got an embassy in China and enjoyed good relations, regularly trading, exchanging innovation, and helping with Muslim settlement in the borderlands. This, with its security of the Silk Road and thus major eurasian trade, was probably much more desirable than what would definitely be an impossibly long, hard-to-continue military conquest across Central Asia into China.
^ China and the Middle East: From Silk Road to Arab Spring