Was it just a matter of India being a loosely organized collection of different cultural groups and the UK being united? Was it a matter of technological superiority? Why did it take until 1947 for India to receive its independence?
This is a good question! I haven't got much time today, so I'll just give a brief overview. But in brief, it's a lot less about military technology than you'd think (and as /u/RedDragon777 implies).
The Context of British Conquest: The Mughal collapse
South Asia was one of the few stable and prosperous regions of the seventeenth-century world. The major population centers of North India flourished under the control of the powerful Mughal emperors, whose dominion was by far the most powerful state that India had ever seen so far. 30% of the Ganges Basin and Bengal were occupied by farms in 1600. By 1700, better breeds of crops and new agricultural technologies combined with the security and stability provided by Mughal rule had allowed 50% of the land to be farmed, an increase of two-thirds. The population generally rose from around 145,000,000 in 1600 to 175,000,000 in 1700. Urbanization and commercialization progressed apace, thanks to increased overseas demand for Indian manufactured goods, the coming of productive American crops and advanced Persian technologies, and a great influx of silver that lubricated the economy (India produces almost no gold or silver).
Meanwhile, Europe and China were tearing each other apart in brutal civil wars.
It's no surprise that seventeenth-century Europeans, British or otherwise, could not imagine possibly conquering the mighty Mughal empire. Sure, they might annoy the Peacock Throne by engaging in piracy or printing fake Mughal coins. But full-on conquest was out of the question. Indeed, in 1689, John Child, English governor of Bombay, tried to further monopolize Indian trade and captured several dozen Indian ships sailing to the Mughal port of Surat. In the brief war that ensued, the English were expelled from their old factories in Bengal, Mughal agents occupied most of Bombay, and the English were forced to prostrate themselves before the emperor and pay hefty war reparations. The East India Company, the organization which would later subjugate all of India under the Union Jack, was still no match for the empire.
But in the early eighteenth century, the Mughal empire disintegrated due to internal factors. The British themselves had little to do with Mughal collapse; the big factors were all internal to India, important ones including a weakening central control over both rural elites and provincial governors, the emperor Aurangzeb's foolhardy decision to overextend resources by engaging in wars in South India, and tensions created by rapid economic growth that often benefited groups with only a loose allegiance to the imperial center.
What matters is that Mughal collapse was an extremely messy affair. There were dozens of contenders for power, both within India (Marathas, Sikhs, Rohillas, Jats, Bengal, Hyderabad, Awadh...) and outside it. (the British, the French, the Persians, the Afghans...) The British, already an influential trading group in India during the height of the Mughals, were simply one of many powers in the massive free-for-all following the fall of Mughal imperium.
This is the most important thing to remember about the British conquest of India. The British did not fight a unified Indian empire, not even a "loosely organized collection of different cultural groups" like the Mughal empire arguably was. Nor did they come and conquer a series of geopolitically stable Indian kingdoms. Far from it. The British were the victors of a century of civil war.
The nature of British technical superiority
So why did the British win this civil war? Were the British technologically superior to their Indian adversaries, for example?
In the mid-eighteenth century, yes. Mughal armies were primarily based on mobile cavalry. This made perfect sense in the Indian context, where there were far more horse archers than in Europe (due to Europe's peculiar lack of a militarized nomad population) and where archers could fire more arrows faster than indigenous gunners could fire muskets. Unlike in Europe, where the preponderance of infantry strongly promoted wider firearm usage, Mughal guns and cannons (while still very common, and often individually more effective than European equivalents) were either for sieges or for support for cavalry.
In the seventeenth century, "European-drilled infantry remained the laughing stock of the Mughal rulers." But as Europeans entered the post-Mughal free-for-all, the gun-totting footsoldiers of England and France turned out to be far more effective than the war trains of Indian horse archers. Centuries of gunpowder warfare gave Europeans exceptionally effective and mobile artillery, while flintlock muskets with bayonets and quick-firing cartridges allowed British armies well-drilled in these weapons to fire at much higher rates than Indian archers and make easy work of Mughal-style heavy cavalry. So in the mid-eighteenth century, the British did have a technical advantage.
In the extremely competitive environment of eighteenth-century India, every military advantage was eagerly sought out for. Indian kings quickly mastered the new weapons and infantry tactics, employing European experts to forge new weapons and drill new troops, innovating on new firearms and infantry strategies, even hiring whole regiments of European troops. By the late eighteenth century, Indians had cannons and guns and firearm tactics that were the equals, sometimes even the betters, of those of the British. Consider the case of the Marathas, the most powerful of India's indigenous post-Mughal kingdoms. As historian Randolf Cooper says in The Anglo-Maratha Campaigns and the Contest for India: The Struggle for Control of the South Asian Military Economy,
"[A]ll" the Marathas seemed to possess artillery so powerful that it rendered the holding of British positions in the field as untenable..... [As one British general said,]
"You must by all means avoid allowing him [the Maratha general] to attack you with his infantry. There is no position in which you could maintain your camp against such powerful artillery as all the Marhattas [sic] have. If you should not hear of their approach until they are close to you and coming to attack you, it would be better to secure your baggage in any manner, and move out to attack them. Do not allow them to attack you in your camp, on any account."
[...]
In 1803 quick-firing breech-loading guns were less than three generations away. The Marathas were ahead of the learning curve in putting greater tactical emphasis on artillery firing huge numbers of anti-personnel projectiles. Western historians are uncomfortable with the idea of South Asian leadership in military science but that is what Maratha artillery doctrine represented. Using a greater volume of artillery fire to dominate the battlefield and increase lethality through projectile selection meant being a step closer to the industrialization of killing.
The British had no notable technical edge on the battlefield while they were conquering the majority of India.
So why did the British succeed?
First, let's look at the details of the conquest. In the 1750s and 1760s, the British conquered the area of Bengal, partly due to luck, partly due to the instability of the Bengal government, and partly due to technical and disciplinary superiority which had not yet been sufficiently narrowed or eliminated by Indian artisans. Bengal just so happened to be the single most economically important region of eighteenth-century India, with more human and economic resources to draw from that any other part of India.
Control over the wealth of Bengal gave the British a larger revenue than any individual Indian foe. More money meant bigger armies and more efficient administration. As Clive of India, the conqueror of Bengal, said to the Governor of Madras (another British possession in India):
Bengal is in itself an inexhaustible fund of riches, and you may depend on being supplied with money and provisions in abundance.
When, in the 1780s, the British were pressed hard by a coalition of the two most powerful kingdoms in India (the Marathas and Mysore), the wealth of Bengal sustained the British front and allowed the East India Company to make very little concessions. Later, when the British were at war against the Marathas, the resources of Bengal allowed them to wage war for years across the entirety of India, even as the resources of Maratha lords were prone to become exhausted after a few months of intensive campaigning.
So one answer is that, yes, India might have "a much greater population size" than Britain. But after the fortuitous takeover of Bengal, the British empire was more populated than any of its Indian rivals.
Just as important was British control over the seas. The Mughals did not have a navy to speak of, and throughout the eighteenth century European, and later British, naval power off the Indian coastline was unchallenged. The problem was that most Indian trade happened by sea (the subcontinent is a huge peninsula, after all). With the British controlling the sea lanes, and with indigenous shipping overseas in terminal decline by the 1760s, Indian merchants simply had to associate with the British to properly access foreign markets. This was why, by 1800, the two largest cities in India (Calcutta and Madras) were both British-ruled ports. Indian bankers and capital, too, naturally flowed to these British metropolises. Hegemony over the sea lanes thus gave the East India Company an economic sway over South Asia unrivaled by any other power and greatly amplified the already significant disparity in wealth and resources between the British and their indigenous rivals.
[Continued]