The first assumption here is that the overseas territories were a major contributor to the national economic health of the metropole. This is at the least a contested assumption and has been since the "new imperialism" of the late 19th Century. It would take a book-length reply to fully break this point down, but some broad outlines to consider:

1. The violent extractive colonialism of the late 19th and early 20th Century, best exemplified by the export of wild rubber from Central Africa gathered by indigenous Africans who were forced to do so by agents of King Leopold's trading company and other concessionary companies certainly made Leopold and some other individuals personally wealthy, much as gold and diamond production in southern Africa made individuals like Cecil Rhodes and Alfred Beit and their financial backers wealthy. But much as you might today question the overall economic contribution of the wealthiest Americans to the overall health of the economy, you might do the same for this moment in the past. Peter Cain and AG Hopkins in their survey British Imperialism argue that the major causal force behind the "new imperialism", at least in the UK, was what they call "gentlemanly capitalism"--essentially market speculation by the fading aristocratic class who hoped that by investing in imperial companies they might reinvigorate their flagging fortunes.

2. By the 1920s, the "new" imperial territories held by England, France, Portugal, and Belgium (Germany having lost its colonies at the end of WWI) were not clearly a reliable source of revenue for their home countries overall though they were perceived as vital by metropolitan countries that relied on exports of raw materials from Africa, South Asia and the Pacific whose prices were kept artificially low through the suppression of labor costs. As many scholars have commented, the new imperialism was "hegemony on a shoestring"--colonial administrations spent as little as possible in their territories and most of the tax revenues they received from indigenous peoples were minimal offsets of those minimal costs intended primarily to drive indigenous populations into wage labor rather than to balance the books. When the Great Depression hit, even that level of minimal spending dropped to the bare minimum. This also produced a ceaseless pressure on imperial bureaucrats to find new schemes that would produce profits both to metropolitan industries and revenues to government, but more than a few of these ended up producing additional pressures on administrations due to requiring coercion on colonized population or by failing in some fashion or another. One of the reasons why the rise of nationalist sentiment in the 1930s produced acute anxiety for European imperial administrations was precisely that they could not easily afford to deploy troops on a sustained scale in most of their territories in response to unrest and yet they dared not ignore such unrest. It's worth noting on this point as well that by the 1930s and 1940s, corporations with significant economic interests in the colonies of their home country had already begun to "indigenize" their management and were preparing to operate in a postcolonial era. When decolonization started in the 1950s, most big multinationals with European connections and investments in Africa and Asia like Unilever, Lonrho, Anglo-American, etc. made a fairly smooth transition to operating in newly independent countries without taking many (or any) losses. So I think it's entirely plausible to see colonialism as something which affected the national economic health of imperial powers far less than we might suppose, despite being something which contributed to some interests making enormous profits from the extractive systems that colonialism permitted or established.

3. The most interesting part of the OPs question to me personally, however--and perhaps it is what the OP had in mind--is whether the generation of people from England and France who had been employed by imperial administrations contributed to postwar recessions when the empire shut down colony by colony. (Let's leave Portugal out of it because its colonies don't shut down until the fall of Salazar considerably later.) I think the answer is complicatedly "not much, if at all" for a couple of reasons. First, many of those employees of empire quickly found work as employees with useful post-imperial experience in the new international institutions of the Cold War. They worked for multinational corporations, for intelligence and diplomatic agencies, as academics, etc. I'm not sure anyone has ever done a tight quantitative analysis of this (I should look over the literature on decolonization) but I would wager that the numbers of people employed by England and France in international work focused on former colonies after 1955 would be greater than the number of metropolitan citizens employed by the colonial civil services at their height. Moreover, if anything, employing recent university graduates in colonial civil service between 1945 and 1955 may have offset postwar recessions--this was the era when both countries were trying very hard to retain their empires on new terms of "partnership" and were investing in building infrastructure throughout their empires (as well as deploying troops to fight nationalist or insurgent uprisings as in Vietnam, Malaya, or Algeria).