It's an interesting question and also quite a difficult one to answer - I certainly can't answer the entire question. But as a South Asian archaeologist who does a lot of work on Buddhist sites;

One aspect of it may relate to the polities and dynasties of Early Historic India - the period when Buddhism both became a major world religion, and also the period during which it rather collapsed in India.

First of all it's important to note that Buddhism emerged in what is today Northern India and southern Nepal. Secondly, although recent archaeological excavations at Lumbini (Nepal) have found evidence to suggest Siddartha Gautama was born around the late 7th century BCE, Buddhism doesn't really develop beyond a regional cult until the rule of the Mauryan emperor Asoka (3rd century BCE).

The spread of Buddhism was inextricably linked to the Emperor Asoka's conversion and subsequent promotion of the religion. Asoka also actively sent out missionaries, leading to the adoption of Buddhism in countries such as Sri Lanka. Moreover, at this time Buddhist monasteries also became closely connected to trade routes, and this link played a significant role in the spreading of Buddhism beyond its natal landscapes of northern India/southern Nepal. Indeed, this is why so many Buddhist sites can be found along the Silk Road, particularly in areas such as the Greco-Bactrian kingdom (modern day Afghanistan - see sites such as Mes Aynak) as well as the movement of Buddhism East into Tibet, China etc.).

The Mauryan Empire was followed by the Sunga Empire (2nd - 1st century BCE), and with this shift came a change in the "state" religion. The first Sunga Emperor (Pusyamitra) was a Brahmin, and is often described as deliberately oppressing/stamping out Buddhism - although there is no real evidence of this, and it is far more likely that we simply see a change in promotion, with Hinduisum being promoted over Buddhism. However even here there's no real agreement, and there's undoubtedly still active support of Buddhist institutions, centres etc. by members of the Sunga royal family.

Buddhism then flourished again in northern India during the subsequent Kushan Empire (1st - 3rd centuries CE), particularly under the rule of Kanishka - however, as with the Sungas, a number of religions were accepted and active within the Kushan Empire - including Hinduism.

The next major dynasty were the Guptas - often considered something of a golden age of Indian history. Again, the Guptas were very accepting of Buddhism (building a number of Buddhist monasteries, temples etc.), but were primarily, again, Hindu.

It's essential here to note that Hinduism and Buddhism are not (in terms of core beliefs or practice) hugely conflicting. Indeed, it is entirely possible for a Hindu to follow the Buddhist eightfold path without any religious conflict. Additionally, a large number of Hindus consider Siddartha Gautama to be an incarnation (9th I think) of Vishnu (though this isn't an "orthodoxy"). As a result, much of the conflict between Buddhism and Hindusim was the result of organised religion and power, rather than about religious belief - Sri Lanka is a particularly good example of this clash.

It's also extremely important to highlight that all of the above only really applies to northern India - southern India saw an entirely different succession of dynasties and kingdoms. The Satavahanas existed contemporaneously (as feudatrories) with the Mauryan Empire, and promoted both Buddhism and Hinduism side by side. However, with their decline in the 3rd century CE, close to a millennium of Tamil kingdoms (Pallavas, Pandyas, Cholas, Kalingas etc.) all of which were Hindu.

(1) Buddhism was a reactionary movement/philosophy/religion opposing the segregation imposed through the caste system. Buddhist teachings center on attaining enlightenment through your own efforts, rather than the assistance of intervening holy men (the highest caste in Hinduism, Brahmins). However, the practice of Hinduism (loosely used in this case, because the term "Hinduism" was not formalized until the arrival of foreigners who needed a means of understanding the local faith/philosophy) was already too entrenched in Hindu society. Brahmins were strongly opposed to Buddhism and sought to steer people away from this "newer" faith. Also, the Mogul/Mughal upheavals of the time did not exactly make it easy for Buddhism to flourish.

(2) Buddhism as it was initially envisioned/taught by Gautama Buddha was too difficult to truly follow. It requires complete separation from desire in order to end suffering (i.e. The Four Noble Truths). The reason Buddhism managed to survive and spread is because an "easier" version became available: Mahayana Buddhism, or the Greater Vehicle. This form of Buddhism incorporates prayers and intercession by bodhisattvas, and is a result of Buddhism mixing with native beliefs in Northeast Asia. It is the popular form of Buddhism known to many today. The original form, however, was simply too difficult for it to gain any kind of traction.

To give you a brutal, hard-to-digest, slightly biased yet true answer as a Brahmin - the Brahmins, who have always been a dominant community in Indian history were against Buddism (or any other religion). Brahmin rulers made all efforts to keep Hinduism a dominant religion of India and they were successful.

Here is a previous thread on this topic, it may help to answer your question: http://www.reddit.com/r/AskHistorians/comments/w4kk7/why_isnt_buddhisms_influence_stronger_in_india/