This is an interesting question but a hard nut to crack really, and you'll find historians - and other scholars in various related and (relatively) distant fields - from differing movements and schools of thought taking radically different approaches to it. In my field alone (British India) traditional imperial historians (often apologists), Postcolonialists, advocates of Subaltern Studies and those of New Imperial History - the latter of whom have greatly influenced my own own work - all take differing view of this question, though certainly all would provide fairly nuanced answers. I think it is equally important to understand that within a single empire various colonial subjects would be treated very differently - often to a far greater degree than it can be said that different empires treated their subjects differently.

To summarize things in layman's terms from what I have learned from my training (in history and political science), a truism of sorts is that while all empires are fundamentally exploitative the nature of said exploitation differs greatly, and the long-term results of it differ even more, leading many theorists to conclude that some exploitive systems - despite their similar natures - benefit their own victims, at least in the long term, better than others.

You bring up one of the most widely discussed examples of this in your question when you mention Hong Kong - a former piece of the British Empire. It is simply obvious today that people from former colonies of the British Empire are better-off today than subjects of other European colonial empires, and indeed some have sought to establish a rough - and usually flawed hierarchy of sorts in this regard with the British at the top, through the French, Dutch, etc etc, to the horrors of Portuguese, Wilhelmine German, and Belgian African colonies.

But this question is more complicated than it seems. Did colonial populations under British rule flourish more readily everywhere the British Empire ruled? Certainly not. By the 19th Century 'settler colonies' like Canada, Australia and New Zealand - and to a lesser extent the colonies of the South African Cape - obtained a high level of colonial autonomy which put their citizens (or at least the white ones) on par with British Citizens in many ways. Conversely, the (equally white) enfranchised American colonists a century earlier had to fight for basic control of taxation, eventually leading to independence. From this one can clearly see that exploitation of colonial populations - even among privileged settler groups was not always equal even within such groups in the same empire at different times and places. Moreover, disenfranchised groups in these areas (ie African slaves, Native Americans, Catholics, etc) occupied a worse position in either case, and of course things were different in other parts of the Empire (again, South Africa comes to mind).

Taking British India as a case study, the British also viewed different non-settler (ie indigenous) groups differently and treated them in different manners, dividing them by such diverse criteria as religion, race/ethnicity, and caste (socio-cultural-economic). Regions, communities, networks, and even trades dominated by different groups would be exploited or suppressed - or conversely protected and promoted - according to the whims of the Imperial power and its agents both as a result of predispositions (prejudice, ideology, etc) and more pragmatic considerations (political necessity/convenience, the desire for profit, etc). A good example of this is how rural North Indian caste-Brahmins played a prominent role in the East India Company (EIC)'s Bengal Army prior to the 1857 Rebellion, and how they were subsequently pushed out of the British military in India in favour of the so-called Martial Races drawn from more marginal groups as a result of the prominent role they played in the rebellion. Here again, we can see how the British treated one group differently at different times, in this case moving from a favoured position to one of disfavour.

Taking this into consideration when thinking of the diverse British Empire one may also consider the relative prosperity of former British Colonies today vis-a-vis the former possessions of other European powers, with some people attributing their relative prosperity today to the British having treated their colonial population better as alluded to above. I am inclined to challenge this view personally, citing India-centric cases of gross exploitation related to my work like the decline of the Indian textile industry and the infamous 1769-1773 Great Bengal Famine, and instead citing both somewhat Whiggish causes for these modern differences like the relative maturity of British liberal democratic and economic institutions at the time of initial imperial expansion and eminently practical ones like the continued utility of English in a still American-dominated world (a strong example of this being how until recent decades many native Cantonese speakers I know have done far better for themselves as a result of their British-imposed knowledge of English than any understanding of Mandarin or British-influenced system of values they may have had).

As with everything, it depends. Native people were often subjugated, but if they had something unique or valuable to offer, they were traded with and treated relatively well (for a time).