From the coast of SE China, to get to Bengal, you have to go through the jungle-covered hills of Yunnan, Vietnam, Laos and/or Myanmar. Not great for marching giant armies. Alternatively, you can cross the Himalayas, which of course, isn't practical. Now, China did invade Vietnam frequently, but it wasn't at all practical to march giant armies to India. Nor was there any particular reason to. For China, the main expansion route was west. West was the Silk Road. West was barbarian hordes who needed to be subjugated lest they attack China once again. South? Mountains and jungle full of (as far as the Chinese were concerned) backwards poor people.

I'll also mention that while it's very difficult for the West (as in, Western Europe) to invade India, it's much less difficult for people in the Afghanistan region to invade, because the Pakistan/Afghanistan mountains are much easier to cross than the Himalayas. Hence everyone from Alexander to the Mughals coming that way.