I've written about this before, but it was long ago. There is quite a lot of contention over the reasons that Buddhism declined so strongly in india.(as an aside, talking about the subcontinent as a whole is a problem. Think of it more like a series of kingships with ethnically and linguistically different people sharing certain aspects of a common culture.)The main key to remember is that religion and social status were deeply tied, with the Brahmin Hindu upper class having a vested interest in maintaining Vedic traditions (and thus their status) and the lower castes often adopting different systems of belief as a form of social commentary. If you think about it, Buddhism in its early stages was (in part) a countercultural push, rejecting much of Vedic tradition and advocating for a radical idea: enlightenment outside the traditional strictures of religious order. However, as time went on and it got more institutionalized monasteries became centers of education and wealth through the sale and trafficking of texts, as well as sponsoring trade expeditions. This means that those institutions started to play an active role in society, tying people together.

In one sense the center of Buddhist dynamism traveled East, into China. Certainly we see texts start moving from China-->India as much as the other way around. If we accept Jan Nattier, even the Heart Sutra was written in China and then translated into Sanskrit. Chinese philosophy and the flowering of Mahayana traditions meant that the hardest textual work was being done there. For more info on China and the beginnings of its Buddhism you might check out :

Gregory's "Doctrinal Classification in the Hua-Yen Tradition"

John McRae's "Religion as Revolution in chinese Historiography: Hu Shih on Shen-hui"

Kenneth Ch'en's History of Chinese Buddhism

Unfortunately there isn't a great single entrance point for understanding Chinese Buddhism, so these are all fairly technical suggestions.

Back to India, one thing we see is the beginning of a fusion of Hindu and Buddhist thinking. Particularly transgressive Shivaism starts to merge with the Tantric traditions. When I say "transgressive" I mean that in the most emphatic sense. David Snellgrove writes (of Tantrism in India) "The "spiced food" of the sacrament refers to a concoction of the flesh of a human being, a cow, an elephant, a horse, and a dog." There are very solid theological points behind practices like this, but as they began to spread they didn't really help the popular image of the Buddhist in India. That, combined with the constant powerful draws towards the dominant forms of Vedic life meant that Buddhism was always going to struggle. Also remember that a 3rd century Indian man will have a very different view of religion and what it constitutes than we do today. There's no real reason why a person could not have worshipped Hindu gods and the Buddha equally/simultaneously. Certainly Hinduism makes ample space for unlimited deities, and Shakyamuni along with the other Buddhas could be (and were) assimilated with relative ease. Not only that, but the only means of serious study was becoming a monk, which entailed significant social and personal sacrifices.

The best answer is that there wasn't so much a Hindu "revival" but rather a Buddhist "decline" (how we define this would have to be explored and nailed down) and assimilation. Law still originated from the Vedas and the structure of castes was never shaken nationwide by the growth of Buddhism. That meant that even Buddhist kings were supported by the Vedic priesthood who served as lawyers, doctors, and spiritual guides. That is a huge amount of cultural momentum to overcome, and given the incredibly flexible nature of that dominant culture Buddhism was simply brought into the fold. It's also important to note that it didn't disappear entirely, Vajrayana and Theravada both thrived in Tibet and Nepal, with ample contact East- West. As time winds on we see the Muslim invasions and conversions sweeping through and ultimately it just couldn't compete. Think of the difference between religious authority in the three religions. Islam: Become an Imam, openly accessible and a socially respectable position. Hinduism: You've got to be born into it (or marry into it), but for those of that caste it was a position of honor and power. Buddhism: You have to give up caste and possessions to join a monastery where you will have limited political power and less authority. Often you beg for alms in the street, all while living in an intentional community.

Reading about the decline of Buddhism/ birth of Vajrayana:

The Story of Buddhism: A Concise Guide to Its History & Teachings-- Don Lopez (this is fairly accessible, the others are a bit more difficult)

Indo-Tibetan Buddhism--David Snellgrove

Indian Esoteric Buddhism-- Ronald Davidson

Making Sense of Tantric Buddhism-- Christian Wedemeyer

As a final note, I am not an expert on early Buddhism or Indian history. My area of expertise lies mostly with Buddhism coming into the west. I have the familiarity that I need to do my work, but there are people much more versed in Indian history and culture who I invite to correct me/ provide more sources.