Huge, catastrophic downside: periodic famines that killed millions of people, directly attributable to British colonial policies.

There were 3 major famines during the British Raj, the 1876-1879 famine, the 1896-1900 famine, and the 1943 famine. Matthew White, in his book Atrocities: The 100 Deadliest Episodes in Human History, calculated a combined death toll of 18 million people from these famines (they ranked #4 on his list). So how did they happen? I'll briefly cover them one-by-one.

1876-1879 famine: Started after a huge crop failure in much of the subcontinent because the monsoon rains didn't arrive. The local British official in charge of relief efforts made it a priority of "disbursing the smallest sum of money consistent with the preservation of human life." He finally set up work camps where meals were provided in exchange for construction work, but the camps only provided daily rations of 1,627 calories, even less than what the inmates of Buchenwald received in 1944. The British outlawed private relief donations, all the while exporting 300,000 tons of grain while Indians starved. Some of the British political class even saw the famine as completely natural, with the finance minister stating, "“Every benevolent attempt made to mitigate the effects of famine and defective sanitation serves but to enhance the evils resulting from overpopulation." White estimates that over 8 million people died in this famine.

1896-1900 famine: Similar story here. A massive drought in 1896 led to a huge crop failure and the British did almost nothing to alleviate it. The Famine Fund that they set up to prevent a repeat of the 1876 famine was quietly raided to pay generous pensions to retirees in Britain. Once again, they exported the previous years' grain surpluses back to Britain. Millions died. The famine was so severe that the 1905 Indian census actually recorded a population decrease from the last census of 1895, the first and only time that has happened since they began taking censuses in India.

1943 famine: The direct result of the Japanese occupation of Burma (a major food supplier) in WWII, this one is harder to blame on the British, but their response exacerbated the suffering. They commandeered all civilian transportation for military use, making it harder to send internal food sources to the affected Bengali regions, and refused to redirect international grain shipments to Bengal. Churchill callously asked that if the famine was so bad, why wasn't Gandhi dead yet?

The common thread running through all of them is the Victorian-era belief that governments shouldn't interfere with market forces, even when the market wasn't providing basic levels of subsistence, and raw colonial exploitation. India hasn't had a famine since independence.

Sources:

Matthew White's Atrocities: The 100 Deadliest Episodes in Human History

Mike Davis' Late Victorian Holocausts