They weren't really "let go" from the British Empire; their status gradually just kept evolving over time.

From the start they were somewhat different in that they were more heavily settled by the British than other colonies (e.g. the ones in the Caribbean, Africa and India). Since the purpose of the colonies in Canada (or "Newfoundland"), Australia, New Zealand and South Africa wasn't strictly economic like the other colonies, but also social (i.e. creating permanent homes for large amounts of British settlers), they were held in a somewhat different regard. Basically, the British government didn't just see these colonies as bread baskets or moneymaking machines where the natives were to be exploited for financial gain, but as a sort of half-way between a colony and an extension of Britain. Much of this had to do with the prejudices of the time; the more white, European settlers there were in a colony, the more civilized it was, and thus the more worthy it was of being given special privileges.

In the mid-to-late 19th century, the most heavily settled colonies - Canada, Australia, New Zealand and South Africa - were 'promoted' in a sense from being mere colonies to being "dominions", or more specifically "the White Dominions". This special status - granted because of the large proportion of white settlers, mostly British - gave them privileges like dramatically increased autonomy. The Dominions were allowed to have their own representative chambers of government, and over time these chambers gained authority and jurisdiction over everything except the most major and large-scale issues like declarations of war and peace, and alliances. The major reason for these huge concessions was the desire of the British government to avoid a repeat of the American War for Independence, which occurred specifically because the Thirteen Colonies were not granted as more autonomy as they wished. During the early 19th century there were rumblings in the Canadian colonies of rebellion because of similar reasons. The initial response of the British government was to crack down on them and suppress them, but wiser minds prevailed and more autonomy was in due course granted, which averted another possibly rebellion. The success of this move convinced the British government that granting autonomy to certain colonies was preferable to repressive action which could provoke rebellion.

This process was not instantaneous and occurred over almost a century, however, from the mid-19th century to the mid-20th century. The Dominions were still officially considered part of the British Empire during both World Wars (the Australian army raised at the beginning of the First World War had the word "Imperial" in its name, denoting its status as a wing of the British armed forces), although by this point they functioned in reality almost as independent states; their immediate declarations of war on Germany were mostly due to their emotional attachment to Britain rather than to any legal obligation. As an example of just how autonomous the Dominions were at this point, when the British Prime Minister requested they send troops for a campaign to block the Bosporus Strait against the Turks during the Greek-Turkish War, all of the Dominions either refused or ignored this request, and there was nothing the British could do about it. Despite that, all of the Dominions were eager to maintain good relations with Britain, and none of them wanted "independence", although at this point they had so much autonomy that independence would really just be a formality rather than a fact.

In the aftermath of the World Wars, in the worldwide spirit of freedom, decolonisation and self-determination, all of the Dominions eventually signed documents and treaties that officially severed their legal ties to the British Empire (almost all of which were purely ceremonial and de jure rather than de facto, anyway), but they still remained essential parts of the British Commonwealth, which was the rebranded name of the British Empire (changed from "empire" to "commonwealth" for what could be loosely described as politically correct and euphemistic reasons). All of these countries still remain part of the British Commonwealth, although like everything else, as each year passes their role becomes more and more ceremonial - a Commonwealth games here, an old oath uttered out of habit there. Occasionally there is a flash of real connection between these countries and the Commonwealth, though, such as when the Governor General of Australia - as representing the Queen - dismissed the Australian Prime Minister and dissolved Parliament and called for a complete re-election because of a major political crisis.

But overall the history of the relationship between Britain and the Dominions was peaceful and harmonious. It's a story not of rebellion and war, but of gradually granting more and more autonomy over the years, of reaching compromises and acknowledging the reality of the situation rather than forcing something else. The Dominions were all united in their friendship and support of Britain, especially during its darkest times in the World Wars. In all of these countries - both then and now - there are millions of families directly descended from British families. The relationship between Britain and the Dominions wasn't an "us vs them" scenario, but different shades of "all of us". Comparisons between how the Thirteen Colonies reacted to their colonial masters and the later relationship between Britain and the Dominions show how different the situations were. They belong to two very distinct periods of history, and shouldn't be conflated together.

Sources:

Empire: How Britain Made the Modern World by Niall Ferguson

The Ottoman Endgame by Sean McMeekin