This is kind of a big question and thus merits a fairly long answer so I'm sorry for the late reply, but I've spent a lot of time collecting material for this one and readying the proper response so that it is comprehensive and informative. I'm only an undergrad and my experience is nothing compared to the vast majority of experts here, but I am able to give this a comprehensive answer that is largely in tune with the literature of the day, even if my own opinion is largely based on previous research as I lack the knowledge of Pali, Prakrit, Sanskrit, Classical Tibetan, Persian, and Chinese necessary to analyze the primary sources. So with that out of the way here's a brief explanation of how and why Buddhism declined in India.
It seems strange that the religion that once dominated the Indian subcontinent and had the favor of the first pan-Indian empire is now all but extinct in it's country of origin, comprising a scant .7%, 87% of whom are Navayana Buddhists^1 --a sect founded only 60 years ago for primarily social and economic reasons based on a radical socialist re-interpretation of the classic tale of the Buddha with little to do with historical Buddhism^2-- leaving historical Indian Buddhism at just 0.0061% of India's population. This isn't even the smallest it has ever been. In the 1871 census of British India (discounting British Burma and Sri Lanka though including some of present day Bangladesh, Pakistan was not yet a British Territory so no need to include it in the record) Buddhism was less than 0.00045% of the population^3 .
Broadly speaking, there is no single agreed upon reason for the sharp decline of Buddhism in India. We can roughly pinpoint the major losses to the 4th, 8th, and 12th centuries CE, but the exact reasons are a matter of some debate. R.C. Mitra gives a total of 7 popular reasons to examine as to the cause of the long decline (Additions by myself in italics):
- "Exhaustion" (of the Religion as a whole or the philosophies thereof)
He, as well as anyone analyzing the first, dismiss the first criticism out of hand, but the others beg more examination than just that. These next 6 reasons certainly seem tantalizing on their own, but together they paint a very strong picture of what happened, if incomplete. Personally, though, I find this list somewhat lacking and poorly structured, it leaves out a very important part of the overall picture and has a few redundancies, so I will address this using three major points which I feel better illustrate the decline and its reasons. They are:
#Loss of Patronage
In 180 BCE the last Mauryan Emperor was killed by his chief general, Puṣyamitra Śuṅga ending the long decline of the Mauryan empire and marking the beginning of the Śunga dynasty. Tradition has it that Puṣyamitra was a tyrant, one who hated Buddhism with a passion and that he defrocked monks and nuns, burnt monasteries, destroyed stupas, and even attempted to once and for all destroy the sacred Bodhi Tree but tradition isn't always accurate. In this case at least, it seems like Puṣyamitra doesn't quite live up to his reputation as Buddhism seems to have continued to flourish under Śhunga rule.^5,6 It is possible that the reputation of Puṣyamitra as the great destroyer of Buddhism is authentic, but that he was halted by the power of the Saṅgha, or perhaps he persecuted them on a far smaller scale, perhaps his persecution was entirely authentic and reversed by his successors, or perhaps it was simply the loss of patronage that soured the Buddhists against him.^6 Regardless of the exact reasoning behind his reputation as a hated foe of the Buddhists, Buddhism hardly lost momentum during this period, quite the opposite, despite the fracturing of the Mauryan Empire, perhaps the greatest patrons of Buddhism in Indian history, the rulers who succeeded them seemed to have largely continued their religious tolerance and patronage of all major movements. Buddhism continued to spread outward even into the Indo-Greek lands, where Menander (Milinda in Pali) converted to Buddhism and all his successors became great patrons of the religion, and to Sri Lanka, where it had been established previously by Aśokan missionaries and would become the home of the Theravāda school (the only surviving Nikāya Buddhist tradition). This was also a period of flourishing ideas and interpretations, marking the birth of Mahāyāna and the Bodhisatva Ideal as well as the creation of the Abidhamma texts and the earliest Nikāya Buddhist commentaries.^5
In fact, in the 2nd century CE, Buddhism seemed to shine brighter than ever with the reign of Kanishka the great spreading Buddhism across the Silk Road and Nāgājurna helping to establish the doctrine of emptiness and formulating many of the ideas present in modern Mahāyāna. Buddhism was spreading fast and was incredibly wealthy, statues and stupas were built, people flocked by the tens of thousands to see debates between rival schools. It truly must have seemed a second golden age for Buddhists.^5
Alas, all things come to an end and for Buddhists that was the collapse of the Kushan empire. Following the Kushan's fall India entered another period of small petty kingdoms with rulers vying to become the Chakkavatin (universal ruler) who would unite India. In this period, in part due to the efforts of the Mīmāṃsā school of Vedic thought which had previously reinterpreted Vedic texts and reasserted the importance of rituals within Indian society, the orthodox Brahmanic religion (which may or may not be called Hinduism at this point, experts differ on that) regained traction and the non/anti-Vedic Samaṇa traditions were pushed aside. Out of this religious climate the Gupta Empire rose to power as a firmly Vaiṣṇavist with the Garuḍa being the family crest of the Guptas.^4,5,6
You would think that this would have marked the end of patronage for Buddhism, but it seemed to have only been a minor setback. Rather Buddhism seems to have continued to flourish under the Guptas who continued to support it, albiet less than Vaiṣṇavism or the Bhakti movement. In fact, the last few Gupta rulers (not to be confused with the Later Guptas, an entirely different dynasty) seemed to have converted to Buddhism to a degree, leaving their mark on Monasteries and monuments more extensively than others. Legend has it that Narasiṃhagupta became a Buddhist monk and left the kingdom to his son upon reaching old age.
Of course, by the time of Narasiṃhagupta, the empire had declined and the White Huns were a constant threat to the Guptas in the West, having already invaded twice. Patronage was certainly lost by the kings who took over Gupta lands who more heavily favored one religion or another. By the end of the fifth century the Guptas were but a shadow of the great empire they once were and had all but vanished by the 6th.
It wasn't until Harṣa took over much of the previously Gupta lands in the early 7th century that India once again had a strong central government --at least in the north. Harṣa was originally a devoted Śavite, a practice which he seemed to largely keep to a degree throughout his life, but he later converted to Buddhism and helped strengthen it's hold on people.^6 Of Harṣa's kingdom, Xuanzang said this:
The believers in Buddha and the heretics are about equal in number. There are some hundred sanghdrdmas with lopoo priests. They study both the Great and Little Vehicle. There are 200 Deva temples with several thousand followers.^7
He spoke of Harsha's generosity highly, a ritual that was held honored both Buddha and Śiva where he gave his riches to the people and patronized the monks and Brahmins.^6,7
Harsha's empire was not to last long past his death splitting into many rival kingdoms, most of whom supported the Bhakti movement or Śaivism or Vaiṣṇavism. It was at this time that it became clear that the long divide between the Saṇgha and the laity had began to take it's toll.
The laity were largely disconnected from the Saṇgha, who continued their pursuits in a largely solitary manner. There were few efforts to reach out to the common practitioner of Buddhism with the texts produced in the years since the death of the Buddha being almost exclusively concerned with the monks and their discipline. Simply put, the common people began to feel very isolated from the Buddha and his teachings and to them, it seemed there was increasingly little difference between the Buddhists and the Hindus. Without the kings and wealthy patrons of the Buddhists to link the Saṇgha to the people at large, the institution of Buddhism largely collapsed throughout India because of that divide.^4,5
Buddhism persisted with the support of the Palas in Bengal and it's ancestral Magadhan home as it seems as though their kingdom was an almost exclusively Buddhist one, enjoying a healthy support from both the laity and the kings of the Pala dynasty. But the Pala dynasty began to decline in the 10th century and Buddhism along with it.^5