There's a lot that could be possibly said on this topic but I'm going to discuss one important intellectual influence on British policy towards their colonies in the 19th century: classical economics.

The Scottish philosopher, Adam Smith, in 1776, published The Wealth of Nations, an incredibly influential book in economics, at least the English-speaking world. A large part of this book was an argument for what we now call free-market policies, including free trade. Adam Smith criticised European governments (Spain, Portuguese, France, Dutch, and British) imperial policies in Asia and the Americas, both on moral grounds, but also more extensively (and probably, sadly, more effectively) on the basis that said governments' policies were more harmful to the imperialist countries' overall interest than free trade and a respect for the indigenous peoples' property rights would have been. One key part of Adam Smith's arguments is that rich trading partners are a good thing for a country's economy:  

As a rich man is likely to be a better customer to the industrious people in his neighbourhood, than a poor, so is likewise a rich nation.

And elsewhere: 

In the restraints upon the importation of all foreign commodities which can come into competition with those of our own growth or manufacture, the interest of the home consumer is evidently sacrificed to that of the producer.

Smith's argument was picked up by other classical economists such as Frederic Bastiat - who had a vivid line in sarcasm, see for example his Candlemakers' Petition, in which he imagines candlemakers petitioning the French government to ban competition from the sun, using the same arguments that other manufacturers used (and still use) to argue for government protection.

In light of this intellectual influence from economists, as well as the obvious failure of British policies that led up to the American Revolution, British politicians had reasons to see their country's self-interest in free trade with prosperous colonies in Canada, Australia and New Zealand. 

(Note: this view of imperialism as a harm both to its immediate victims and the imperial country's economy is controversial within economic historiography.  But even if Smith was wrong, his views were still influential.)

So, given all this, you might well be wondering why Britain didn't grant colonies like India or Ireland home rule as it did in Canada, Australia and NZ, and why did Britain even acquire more colonies in the 19th century, for example in the scramble for Africa? 

There's a multitude of reasons, few of them happy. In no particular order:

Firstly, one political pressure for imperialism is that, while imperialism might be bad for the home country's economy, it can be very beneficial for the actual colonial authorities. A number of colonial administrators got very rich (famously Clive of India) and more hoped to get rich, and therefore had incentives to lobby against self-rule in those countries. In the case of Britain's colonies formed pre-19th century, such as India, these interest groups were already in place. There's a well-established branch of economies called public choice theory, which is about understanding how real-world governments make decisions, often ones that are not in the public interest. One result of public choice theory is "concentrated benefits, distributed costs" - if a policy gains a small group say £10,000 each, while costing every voter say a few pennies, the small group has a strong incentive to lobby for it, while it's not worth an individual voter's time to lobby against it. 

Secondly, there's also some free-trade arguments for some imperialist activity: e.g. British involvement in Egypt in the context of the Suez Canal. Niall Fergusson argued that British empire was beneficial as a way of ensuring free trade within it, and in favour of his argument not all independent countries were as willing to trade in the 19th century as the USA, consider for example Japan's self-imposed isolation. 

Thirdly, and in some contrast to my second point, classical economics never acquired universal acceptance in Britain in the 19th century. There's always been a line of argument that colonies were economically beneficial to Britain, most famously by Marxists and Marxist-influenced intellectuals, but there's a lot of "folk economics" about and also some more considered arguments for tariffs, such as infant industry arguments.

Fourthly, there are also non-economic motives, such as pride in living in the largest empire the world has ever seen: far surpassing the Romans British gentlemen studied at school. To quote the economic historian Deirdre McCloskey:

Queen Victoria loved being an Empress and Disraeli loved making her one, so imperial India happened.

Finally, and miserably, there's also bigotry and racism: the 19th century saw the growth of 'scientific' racism. There was a shift amongst British intellectuals such as John Stuart Mills to regard European civilisation as superior and other groups as incapable, or at best, unready for self rule. 

In summary, in Britain in the 19th century there were intellectual influences favouring a better treatment of colonies, which benefited Canadian, Australian, and NZ settlers. Other subjects of British colonies however were exposed to greater extractive pressures by colonial officers and British racist prejudices. 

So, that's one aspect of why. Now I have my biases, in particular I am a NZer born and bred so I know more about NZ history than Australian, and I know very little about Canadian history. I understand that colonial Australia tended to have a more fraught political relationship with Britain than NZ did, understandably given convicts and Irish influences, and there's probably a lot to be said about the political ins and outs. History is contingent, after all. 

Sources: (partial)

Acemoglu, Daron; Johnson, Simon and Robinson, James A. “The Colonial Origins of Comparative Development: An Empirical Investigation.” American Economic Review, 2001, 91(5), pp. 1369 –1401.

Public choice theory, http://www.econlib.org/library/Enc1/PublicChoiceTheory.html Pitts, Jennifer (2005). A Turn to Empire. Princeton, N.J.: Princeton University Press.

  Tillman W. Nechtman. A Jewel in the Crown? Indian Wealth in Domestic Britain in the Late Eighteenth Century Eighteenth-Century Studies, Vol. 41, No. 1 (Fall, 2007), pp. 71-86

William Easterly: The White Man's Burden: Why the West's Efforts to Aid the Rest Have Done So Much Ill and So Little Good, Penguin Books, 2007.

Deidre McCloskey, "Keukentafel Economics and the History of British Imperialism," South African Economic History Review 21 (Sept 2006): 171-176.

Niall Fergusson, Empire: How Britain Made the Modern World, 2003