There were a number of factors that led to the decline of Buddhism in India, but the way the question was stated implies that Buddhism was - for a period - the most popular religion in India (1). This is not true. Of the native Indian religions the Vedic / Brahmanical religion, and its modern permutation of Hinduism, has always been much more popular than Buddhism. Buddhism and other heterodox religions like Jainism (and to a much lesser extent the materialist Carvaka school) were minority philosophies in India. They appealed to particular demographics. For example, during the Buddha's era, it is thought that Buddhism, as a lay activity, was popular amongst the urban merchant middle class. Because Buddhism appealed to a particular group of educated merchants it was not always going to be sustainable. It was always going to be a minority religion.

Another point here is that the significance of Buddhism in even the most important period of Buddhist power in India, during the Asokan dynasty, has been questioned by scholars. Many people believe that the Mauryan kingdom of Asoka was a defacto Buddhist state because Asoka was a self declared Buddhist. But the sincerity of this Buddhism can be questioned in a number of ways. For starters, one of the Asokan inscriptions that is commonly attributed to Asoka's purported Buddhist motives is in fact a direct lift from Jain textual sources. Another inscription again associated with Buddhism is a Brahmanical text. There are other reasons to doubt Asoka's Buddhist credentials and although we have no reason to doubt his conversion it isn't clear that his kingdom could be described as a Buddhist one. It was almost certainly pluralist and Asoka may have supported, or even followed, several different religions.

Fa Hien (5th ce) observed in his travels of India that while Buddhism was popular in some areas, in areas of traditional importance to Buddhism such as Gaya and Kapilavastu, Buddhism was very much in poor condition.

Hiuen-tsang, a Chinese Buddhist traveller who visited India in the 7th ce observed that already Buddhism was in decline and a number of temples and monasteries had been abandoned. He observed that the Brahmanical / Vedic faith was popular.

So it is important to not think that Buddhism was ever dominant anywhere in India.

Regarding the collapse of Buddhism: many people point to the Muslim invasions as a direct cause of the end of Buddhism. This is only part of the story. The Muslim invasions helped accelerate the end of Buddhism which was already in decline. The Muslims destroyed many Buddhist shrines and learning institutions. I believe the destruction of Nalanda university was due to muslim invaders. So those invasions did not help matters at all. Anyway, one of the reasons you hear so much about the Muslim invasions as a factor is because this argument is often articulated most forcefully by Buddhists themselves. Naturally, there is a tendency to overlook some of the internal problems Buddhism in India had that led to its collapse and exaggerate the importance of external factors.

As I said, Buddhism was already not in great shape.

• In the 8th-9th ce in South India there were vigorous anti-Buddhist crusades by Saivite and Vaisnite scholars against Buddhism. There were sometimes violent attacks against Buddhists. This led to Buddhists practically being expelled from certain regions in SI.
• In the South Buddhism was ultimately taken over by Jainism and Saivitism.
• In some parts of the North of India and modern day Pakistan Buddhism lived side by side with other religions until the 12th ce. The spread of Islam in Kashmir had the effect of forcing out rival religions. Likewise, in central and western India, the spread of Islam helped lead to the demise in Buddhism. Muslim persecution was therefore a factor as stated.
• Throughout many parts of India the popularity of tantricism helped lead to a collapse in Buddhism. Tantric practice began to influence Buddhism to such an extent that it practically because synonymous with Hinduism.
• But perhaps one of the biggest factors in the decline of Buddhism was the increased perception that the Buddhist order had become corrupt and fallen away from their religious commitments. Buddhist temples became centres of wealth and hubs of finance. It is also reported that monks and nuns engaged in improper conduct in violation of the Buddha's basic teachings. Furthermore, sectarian divisions tore apart the unity of the religion as various early Buddhist traditions vied against one another while at the same time competed with Mahayana Buddhism. Monks just weren't acting like proper Buddhists. Lay people began to turn away from Buddhism towards more stable religions.
• India has also always been very eclectic in its religious traditions and believers often take a syncretic approach to religious belief. Creating a narrative that allows the inclusion of other religions is normal, but this isn't always good for the viability of a religious tradition, especially a tradition like Buddhism that has always been a minority.

(1) I would also like to point out that the term 'India' makes no sense during these ancient and medieval periods. India, as a state, is an invention of the colonial era. You may know that already but I feel compelled to add a note here to state that outright.

References:

• Rise and Decline of Buddhism in India by Kanai Lal Hazra

• Theravada Buddhism: A social history from ancient benares to modern colombo by Richard Gombrich

Edit - confused Mughals with earlier Muslim invasions.

First, we need to keep in mind that "Hinduism" is wildly misunderstood as a concept. The indigenous religion of India, which is properly called "Brahmanism," derives its authority from the priestly Varna called "Brahmins." This Varna structure was well entrenched in Indian society by the 7th Century and while the Brahmanists followed this line of authority, Buddhists did not. One of the Buddha's more famous (albeit misunderstood teachings) was that proper Varna was not necessary for enlightenment. Previously, it was believed that only Brahmins could be enlightened. It is commonly believed that the Buddha spoke out against the Varna system, but in actuality, he said that anyone - slave, merchant, warrior, priest, or outcast - could be enlightened. Society outside of the Sangha maintained its social structure which did not necessarily alienate Buddhism, but - as is evident in today's Buddhist societies - Buddhism was not strictly divided from Jainism and Brahmanism on the subcontinent. It's reasonable to believe (again, we can point to examples of Buddhist societies in Tibet, Bhutan, China, southeast Asia, Korea, Japan, etc.) that Buddhism existed side-by-side with local beliefs and traditions.

The easy answer for why Buddhism "slowly" declined from the Subcontinent would be the arrival of the Muslim invaders. The Muslims say only idolatry when they arrived in India and indiscriminately burned and destroyed as much as they could of the Buddhist infrastructure in India, one of the greatest losses was the University of Nalanda. Priceless scriptures were already being translated and taken to Tibet where they survived until an ironic 1959 reversal of this trend, when Chinese soldiers began lining their boots with sacred Tibetan scriptures and thousands of documents are (still) being smuggled out of Tibet to India. Today, in Bodh Gaya, there is an ongoing effort to translate the Kangyur and Tenjyur into their original Sanskrit from surviving Tibetan sources, but I've gotten a little off-track here.

The Muslims, for centuries, existed with the goal to convert the entire Subcontinent, but they were woefully unprepared with how difficult of a task that would be even though they did succeed in destroying much of the Buddhist infrastructure and defeated many Buddhist patrons in battle. The last major Buddhist dynasty in India was the Palas, a Bengali-based dynasty that much of the current Tibetan system and philosophy was based off of before they escaped to the north. Once the Muslims were done rampaging through the region (and dramatically reversing the religious trend in the Bengal) there was never as high ranking of a Buddhist leader on the Subcontinent to support the Sangha and it largely disappeared in its major forms to the Himalayas and Southeast Asia. That's not to say that Bodh Gaya, and other Buddhist pilgrimage sites, haven't maintained a powerful Buddhist presence ever since the Tathagata's enlightenment.

That said, Brahmanism is not immune to change. It is a very fluid religious complex which changes quite often. One of the most recent changes was the adoption of "Hinduism" which is a political term that directly references the "Indus River" and is used by (and I may be the only person to be using THIS particular term) "Brahmanist extremists" who believe that just as Pakistan is for Muslims, India is for Hindus (both reference the Indus River... which is in Pakistan, an irredentist claim).

That said, Brahmanism absorbed many elements of Buddhism after living side-by-side for so long. Vaishnaivism, one of Brahmanism's larger denominations, actually references the Sakyamuni Buddha as one of the ten incarnations of Vishnu. I'm not as well versed in Vedic philosophy as I am in Tantric Buddhist philosophy, so I wish I could elaborate more, but I suggest on checking out some books on the Muslim invasions of India, Buddhist history in India (Andrew Skilton's is pretty good).