There are many reasons, but I'll just mention the rather noticeable geographical barrier, the Himalayas. They are the tallest mountains in the world, and it would be hard to lead a large army across.

For much of history, India didn't even border China in the north, it bordered Tibet. Tibet was briefly under Chinese control back in the Yuan period (when both China and Tibet were conquered by the Mongols) and more recently under the Qing Dynasty (1724-1912), a period of 188 years. Plus, of course, the modern takeover of Tibet by Communist China in 1950. So for most of the past couple thousand years of history, there was a big buffer kingdom between India and China, which was Tibet.

In the same way, there was a buffer kingdom on the east, which was Burma. Also, the Naga Hills extend pretty far south along India's eastern border, and while they are not as tall as the Himalayas, they are densely forested tropical jungle, which has historically supported only scattered tribes of headhunters. Again, it's not the easiest terrain to lead an army through.