This question is impossibly huge because the Empire was massive and most cases had specific developments that may or may not be considered 'loss.' Decolonization studies is a going concern, from D. George Boyce (1997?) onward, so the process has its historians. Dane Kennedy wrote Decolonization: A Very Short Introduction for the Oxford series in 2016, and Jan Jansen and Jürgen Osterhammel wrote Decolonization: A Short History for Princeton in 2017 most recently.

Two things may surprise you. First of all, it's questionable whether one may call it "England." I assume you are discussing the British Empire, so England is only in play until the various Acts of Union, at which point Great Britain and then United Kingdom become correct (although we tend to use Britain as shorthand). Second, if we consider the settler Dominions (including South Africa) to be still part of the Empire clear until the Statute of Westminster, then the high-water mark for the British Empire is about 1930, at which point the Dominions have enough legal power to chart a different course although they tend not to. The end of empire turns on a lot of factors, different in almost each case, and honestly each requiring its own discussion. I would encourage you to look at the Empire circa 1930 and ask about particular territories or regions.

But what is 'its territory?' When does it remain, or cease, to be British imperial territory? Is it no longer strictly British territory when there is meaningful local self-government, or does it persist when there's still a major political link even if not truly prohibitive of separate action (like the British Commonwealth)? These questions of legacy and control are not mere semantics, but exercised a variety of opponents of empire around the world who took very critical views of colonial and neocolonial relationships across the various regional waves and particular cases alike.

I'd suggest you start with the short histories and perhaps Boyce's book, and for the high water mark of European empire generally, Matthew Stanard has a new book out (May 2018) on European empire from 1879-1999.