There's an instrumental battle for this period, the "Battle of Talas" or the Battle of the Talas River where the Chinese and Abassid Empires met to battle over Central Asia. This was pivotal in defining "regoins of influence" i guess if you could call it that back in those days. It begun to slow Islam's expanse into China but significantly slowed China's expansion into Central Asia.

Also let's just also take into account the distances we're talking about here. I mean with the distances you're talking, with the stability of the Abbasid Empire, and the resistance of an established state which already proved it could provide a armed resistance at so far a frontier in Talas (nowaday Kyrgyzstan), it's not too surprising that the Abbasids weren't able to penetrate China via some of the harshest terrains over such a long distance.

I guess to give you some basic but not necessarily helpful data, according to Google Maps, by road, Damascus to Edinburgh in Scotland is just under 5,200km. From Damascus to Xi'an which was probably the most accessible western city in China is over 8,200km. I know the situations are different between them but this is just to give you some kind of scale of why I'm not sure you could have just expected a Muslim invasion of China based on expected proximity.