(part 1/2) I am writing a dissertation about British imperialist ideology up to 1901, so I hope I can offer some insight! (I have to say, I'm always excited when I see a question on AH that makes me think "I KNOW A LITTLE ABOUT THAT!") Short answer to whether the British believed the Empire would be forever: both no and yes. Practically no, ideologically yes, and this was very much a contradiction. Slightly longer answer: They knew that sooner or later it had to come to an end, but the end of the Empire and even more so what happened after the end was, for ideological reasons, almost impossible for the British to entertain. As for when the British acknowledged that independence was inevitable, that is also very important, but I will focus on the first part due to limited time. If I have time later, I will address the second question. I will answer the first by invoking different politicians' ideas about India, looking particularly at the school of thought which sought to restrict Indian Home Rule as much as possible, and only granted concessions when to not do so would have led to revolution. This school of thought was in the majority during the time we're talking about.
I think it's first important to first talk about dates. You mention the 1920s, but I will start before that (partially because that is my primary area of research, though I can talk about later, too). The foundation stone of New Delhi was laid in 1911, at the Delhi Durbar, when King George V was proclaimed Emperor of India at Coronation Park. The date is important because, while New Delhi was mostly built in the 1920s after the First World War, its purpose as the new colonial capital was set out and declared amidst much pomp in 1911. This was before the horror and expenditures of the First World War, including the unrest that service in the war caused for troops from British India (an administrative region including, among others, what is today India, Pakistan, and Bangladesh).
It was also before the Jallianwala Bagh massacre, known to the British at the time as the Amritsar Massacre, in 1919, when according to the British report 379 Indians were killed and many more wounded, while others have estimated over 1,000 dead. This is seen by many as a watershed moment in the history of the independence movement in India, but Kim Wagner in a 2016 article argues that it was the last in a long tradition of brutal violence perpetrated by colonial authorities on Indian subjects, comparing the massacre of unarmed and nonviolent people to a firing squad, invoking Michel Foucault and Stacey Hind writing about the use of public executions to maintain hegemonic power in Europe and in colonial repression, respectively. After Amritsar, arguments for expanding Home Rule increased, and while some felt that this would inevitably lead to an independent India, many still believed India would still be ruled by the British to some degree, perhaps as a dominion like Canada or Australia (the degree of autonomy of a dominion would itself change in 1926 with the Balfour Declaration, and India was briefly a dominion from 1947-1950). The Government of India Act of 1919 allowed India partial home rule, with George V declaring that it "points the way to full responsible Government hereafter."
My point is that the situation in India changed dramatically between 1911 and 1920, but one thing did not change: the British determination to rule India for as long as possible, despite tacit acknowledgement that the British Empire would, one day, come to the end. The nominal symbolic meaning of New Delhi, however, would shift because of the political changes, and I will touch on that at the end.
Next, a brief note on what India meant to the British imagination, and then we'll move to looking at how the British talked about the possible end of the Empire. To the British, India was famously "the most beautiful jewel in the British crown," but even beyond that, India had become a part of the way the British fundamentally saw themselves. Post-colonial theory has as one of its fundamental tenets the idea that the identity of colonizer and colonized are hybrid (each is in the other) and interconnected. The subject position of both is constructed by its relationship with the other in a process termed intersubectivity. As described by Frantz Fanon in The Wretched of the Earth in 1961, the colonial relationship has a profound psychological effect on both the colonizer and the colonized. The colonizer internalized the position of power. In a case study, Fanon describes a colonizing agent whose task it is to torture those under his power. This agent also beats his wife. He feels guilty about the latter and seeks help. When he turns to a doctor, he asks for a way to torture without feeling guilt. If he did not feel guilty about hurting the colonized, he thinks, he would not take his guilt out on his wife. He never thinks to seek a different career or reflects that maybe his guilt about what he is doing to his wife also suggests a repressed guilt over what he does to the colonized. Edward Said in 1978's Orientalism describes the need for the idea of the Orient in order to construct the idea of the "Occident," of Europe: "the Orient has helped to define Europe as its contrasting image, idea, personality, experience." Robert J. C. Young (1996) shows that the very notion of race (such as of an English race or a white race) requires a dominated, Other race to construct it in an oppositional dialectic: "The idea of race here shows itself to be profoundly dialectical: it only works when defined against potential intermixture, which also threatens to undo its calculations altogether. This antagonistic structure acts out the tensions of a conflictual culture which defines itself through racial ideologies." In a sense, the British needed their colonies to maintain their sense of their own identity, their own relation to the world.
The British attitude towards India has been expressed in far more detail and depth than I will get into here, but I thought it would be remiss not to go into a few highlights. Let's start with Charles Metcalfe in 1815, the same year as the Battle of Waterloo, writing about the imperial destiny of Britain and the future of colonizing and colonized races. Metcalfe was a diplomat in India who would go on, briefly, to be acting Governor-General. He comes just short of saying it is God's will that the British rule the Indians, and insists that what matters is that the British rule well, because if they don't they will be condemned by future generations:
The world is governed by an irresistible Power, which giveth and taketh away dominion; and vain would be the impotent prudence of men against the operations of its almighty influence. All that rulers can do is to merit dominion by promoting the happiness of those under them. If we perform our duty in this respect the gratitude of India and the admiration of the world will accompany our name through all ages, whatever may be the revolutions of futurity; but if we withhold blessings from our subjects from a selfish apprehension of possible danger at a remote period, we shall not deserve to keep our dominion; we shall merit that reverse which time has possibly in store for us, and shall fall with the mingled hatred and contempt, the hisses and execrations, of mankind.
It's worth noting he certainly raises the specter of the possibility that India will fall out of British hands ("whatever may be the revolutions of futurity"), but sticks to the standard and contradictory idea that British rule frees the Indians, "conferring on our Indian subjects the blessings of Independence and Education".
Now, let us move forward in time to the height of the Empire, and turn to the writings of the Under-Secretary of State for India, a man whose name could not be more Victorian, Sir Mountstuart Elphinstone Grant Duff. He, like many, refused to look at the far future as anything but dim and unformed. To imagine an end to the Empire would have been unthinkable; to imagine the Empire as eternal would have been obviously foolish. A very telling passage in his Notes of an Indian Journey in 1876 uses the metaphor of a ship in fog that does not know the future:
I never read a description of a great ship steaming through a fog on the banks of Newfoundland when icebergs are known to be about, without thinking of our government of India. We can do nothing except what the captain does in that case — get the keenest -eyed men in the ship to watch, and go right ahead. It is impossible not to see that things which we are promoting every day in India draw with them very great dangers; but still it is right to promote them.
We return to Grant Duff a little later as, quoted by David Gilmour in The Ruling Caste: Imperial Lives in the Victorian Raj (2005), when he is getting ready to leave his post as Viceroy:
In 1885 Grant Duff drafted a memo for the new Viceroy, Lord Dufferin, setting out the ‘two diametrically opposite views’ on the nature of British rule in India. “One school says ‘You are here to educate the natives to govern themselves. That done, you have only to go about your business.’ The other school says ‘No man knows the secret of the future; but for practical purposes you must act as if Great Britain were to govern India for all time, doing nothing which in your judgement has any tendency to undermine the foundations of British power.’ Grant Duff left Dufferin in no doubt that he was a member of the second school.
The Montague-chelmsford reforms (1919) did promise self government but it was not till the middle of the second world war that it was acknowledged that it would be impossible to hold on to India.Churchill said in 1942 "I have not become her majesty's first minister to oversee the liquidation of the British empire".But within 5 years India achieved independence. Regarding New delhi,the British shifted capital there in 1911 mostly because of annulment of Bengal partition,the need to placate Muslims and to insulate central government from provincial politics. In the 1920s even the Indian nationalists demanded complete independence by 1929 only.British view during that period was that they were going to stay in India for long but with necessary political decentralisation. It was the Quit india movement of 1942 and the mass chaos that it caused, including establishment of pockets of liberated zone that led to British acknowledging that it is getting tough to rule India by force. This trend continued and the revolt by royal navy and general radicalization of army sealed the British fate. One interesting observation a little earlier by Lord Curzon- when we thought of granting self government to Indians we thought of about 500 or so years from now(quote not exact).