I'm not convinced it was entirely a phantom. During the Boer War Britain was isolated diplomatically and the sympathies of Europe were frankly with the Boers. Russia had long felt hemmed in by British opposition to Russian expansion in Central Asia, the Bosphorus, the Balkans and the Mediterranean. Tsar Nicholas visited the German Kaiser to remind him of his previous support for the Boers and he tried to build an anti-British alliance with the French. The Tsar wrote that

...it is pleasant for me to know that I and I only possess the ultimate means of deciding the course of the war in South Africa. It is very simple – just a telegraphic order to all the troops in Turkestan to mobilise and advance towards the [Indian] frontier. Not even the strongest fleet in the world can keep us from striking England at this her most vulnerable point.

Luckily for Britain, Germany and France declined his offer.

On your last question, it was commonly assumed that India was essential to British power. "As long as we rule India," the Indian Viceroy Lord Curzon proclaimed, "we are the greatest power in the world. If we lose it, we shall drop straight away to a third-rate Power". Was this true? I don't think it was.

Firstly, the possession of India led to enormous burdens. Not only the thousands of miles of frontiers to defend, but 450 miles of the North-West Frontier which was in a state of constant readiness for combat. Britain's expansion into South Africa had originally been motivated by the need to control the Atlantic passage to India, as had the purchasing of the Suez Canal. This in turn led to a large Middle Eastern empire too. Much of the Empire was justified on the need to secure the passage to India. If India was the "jewel in the crown", did the crown exist only to support the jewel? In return for these vast strategic commitments and economic burdens, what was India worth to Britain in economic or strategic terms?

India added nothing to the industrial power of Britain. In 1921 71% of Indians worked in agriculture and only 12% in industry. As for raw materials, India only had jute, chromium and manganese. In 1913 India as a market for British exports was worth £70,273,221 (France and Germany combined were worth £69,610,451) but this was less than half the figure for the Dominions (Canada, Australia, New Zealand and South Africa). For British investments abroad, India (at £378,776,000 in 1914) was worth half of America and not much more above Argentina (£319,565,000).

In the twentieth century, industrial power was paramount and just as important as ships and men. This was demonstrated by the experience of WWI and the Report on Indian Constitutional Reforms (1918) noted that "Nowadays the products of an industrially developed community coincide so nearly in kind though not in quantity with the catalogue of munitions for war that the development of India's national resources becomes a matter of almost military necessity". Britain had failed to do this and the Report noted that India's resources "should henceforth be better utilised. We cannot measure the accession of strength which an industrialised India will bring to the power of the Empire". However the industrialisation of India was not something the British would or could countenance. Irrigation schemes were about as far as this went.

India was undoubtedly loyal to Britain during WWI and her volunteer army of over one million was heartening. However India raised just 0.3% of her peoples for the military, compared to 12.4% for Britain, 11.6% for New Zealand and 8% for Canada and Australia. Just 90,000 Indian troops were sent to the Western Front, the rest were sent to defend the Middle East, which concerned Britain only because it was a gateway to India.

On balance, then, India served only to weaken Britain and was a classic example of imperial overstretch. The clue to why Britain remained in India despite this is is due to the nature of British society. Curzon said the Raj was the "miracle of the world...the biggest thing that the English are doing anywhere": India served as a vast arena for the British ruling class. The civil service, the Army, administrators, clergymen, all were needed in British India and the British public schools supplied her with them. As the ruling class was fairly small, everyone knew someone who was or had been in India. The idea of jettisoning India was inconceivable. They did not think in economic or strategic terms but of public service and duty which they had been inculcated in at public school.

In a sense, it sounds as though you have a handle on Britain’s strategic situation in the late 19th and early twentieth centuries. Their absolute priority in imperial policy was always to preserve the Raj, the jewel in the crown of the British Empire. However, as you point out, it was rarely in imminent danger (with the exception, you rightly suggest, of Japan’s campaign in Burma. Rather, Britain mostly played the long game and sought to entrench its position for the longue duree. Much of Britian’s broader foreign policy across the last two centuries was based on securing the long-term survivability of its influence in India, including the possession of the cape c. 1806 and its protectorate in Egypt.

Let’s examine, then, Britain’s imperial anxieties. What were they afraid of with regard to India? For one, they did worry a great deal about growing Russian influence. Their struggle for hegemony against them in Central Asia, immortalized by Kipling, was founded on a few issues. For one, Britain was interested in checking Russian power not only in central Asia but across the globe; they did not want to see a Russia with imperial ambitions upset a balance of power which saw Britain at the forefront and they went to war with them in 1853 in order to halt its growing influence. Russia expanded very quickly into central Asia during the 19th century, and so Britain thought it in its interest to set up its own puppets to secure possible invasion routes into India. Most notably, the British fought several horrible wars in Afghanistan in order to stave off Russian influence in the Raj’s most vulnerable border. It was only after the rise of German weltpolitik and mutual concern over their ambitions in the middle east and Europe, as well as Russia’s terrible performance against Japan that Britain began to worry less about the threat from the Czar in Asia.

The worry from Japan was of a slightly different sort, especially in the twentieth century. Japan was not only militarily and politically threatening, but ideologically threatening. Empire was built on the belief in the innate inferiority of some people compared to others. In Britain, this was expressed most clearly in the anthropological theory of sociocultural evolutionism: civilizations occupied different rungs on the the same ladder between savagery and civilization. Asian peoples were regarded as in the middle zone, moribund cultures still centuries or millennia removed from advanced Europeans. Empire was a mechanism of “uplift,” by which the colonizers were on a civilizing mission to bring progress to the colonized. Japan knocked those racialist theories on their head by suggesting that Asian peoples could not only govern themselves, but that they could form empires of their own. Britain was actually quite impressed by Japan’s prowess in their war against Russia and formed an alliance with them in 1902, the first alliance they had formed since the 1860s. Nonetheless, Japanese expansion would only come at the cost of European influence in Asia, whether on the part of the British, the French, the Russians, the Dutch, or the Americans—and by the interwar period it became clear that a militarized Japan could be a substantial threat to imperial interests. All of these anxieties were absolutely realized in 1942 when the Japanese took Singapore. An Asian army had taken Britain’s key imperial port city, and they did so handily. It is not for nothing that one historian titled his book on the city’s capture the Worst Disaster. While Japanese occupation was hardly welcome in Asia, it did demonstrate that the empire was fragile and that Asian states and Asian people could defeat it. In the case of India, that sentiment was enacted most famously by Subhas Chandra Bose and the Indian National Army, a guerilla organization of tens of thousands that allied with the Japanese and sought the removal of Britain from the subcontinent.

And that leads me to the third, and by far largest, anxiety Britain faced with regard to india: Indians. India was and remains a massive place, home to a plethora of languages and hundreds of millions of people. The British civil service in India numbered in the thousands. The absolute fear was an uprising against british rule. After all, direct rule in India was itself the result of a rebellion that traumatized, or at least traumatized the British imagination. For the entire span of British rule, the unspoken risk of peasant insurgency permeated everything. It was not for nothing that Orwell, in Shooting and Elephant, still on a word-for-word basis perhaps the best thing ever written on colonialism, wrote about the pressure he felt from the crowd, that “my whole life, every white man's life in the East, was one long struggle not to be laughed at.”

Finally, the worth of India to the empire is literally uncalculable. I mean to say that I’ve never seen a convincing figure for its worth in modern pounds that is convincing since any such exact figure would be essentially meaningless across such a long period. However, it made some people astoundingly rich, and those who newly returned from ventures in India were regarded as astoundingly gaudy flaunters of new wealth. India was a site of massive capital investment by the City of London as new railroads were built, making still more people very rich indeed. By the 20th century, fully a fifth of Britains exports went to india—one reason Gandhi so vehemently pushed for a boycott to make his movement felt. And, of course, India was a massive supply of labour, not just for bulk of the empire’s army, but for its construction projects across the indian ocean and the world. The massive indian diaspora today exists in no small part thanks to the huge migrations of labourers across the last two centuries.

Side note, it's "through". Not "threw".