This is a controversial question. There's two parts to it, firstly did colonialism (in the sense of European empires) matter, and secondly, how much did trade at all (e.g. intra-European trade or trade with the USA post Revolution) matter?
In 1776, Adam Smith, in his book The Wealth of Nations, presented an argument that the economics profession has generally found quite compelling: that European colonies were bad for European economies - as well as for of course their direct victims.
Adam Smith's argument has been picked up and refined, the basic parts are:
It is better to have rich trading partners rather than poor, but colonialism is destructive: of physical and social capital and of course of people's lives. Poor people can't produce much to exchange for exports - and of course it's imports that benefit the home economy.
Colonial administrators sought to make large profits for themselves by using their colonial powers to buy cheap in the colonies and sell high back in their home country. This obviously was harmful to the colonial victims, but it also harmed the home country economy by raising the prices that domestic consumers and domestic manufacturers paid for their imports, and thus reducing the money they had to spend on other things.
In the long-term (so over a few years), labourers have to be paid at least enough to stay alive and raise a few kids, the providers of capital (e.g. mining equipment) have to be paid the going rate for their investment, as do the transport systems, so the amount you can save via exploitation versus fair trade is much less than the total price paid under fair trade.
In support of this argument, we do see that Britain and France didn't get poor when they lost their empires in the 20th century, and a number of European countries like Switzerland and Sweden industrialised without having an empire. Most trade nowadays takes place between rich countries: poor countries don't have as much to offer because they're not very productive.
However this argument contrasts a colonial empire with freeish trade, like how the USA traded with Britain after the American Revolution. Of course this was not the only option: Japan famously retreated into (near) isolation until the 1840s. Niall Fergusson made the strongest argument I've seen for the British empire (but I'm grading on a curve), on the basis that it set up and defended a system of free trade.
Deidre McCloskey has a counter argument to this: that trade, all trade, wasn't that important to the British economy. She points out that the increase in British national income was about 100 percentage points between 1780 and 1860, while trade was only about 13% of national product in 1841 - and over that period it moved up and down by about a factor of 2. So only about 13% of the change in income can be explained by trade.
Deirdre McCloskey also addresses "dynamic" arguments for trade: that say cotton textiles was essential for take-off. McCloskey points out that we don't know what potential productivity gains from scale there were in other sectors.
Clark, O'Rourke and Taylor, 2014, pick up on McCloskey's argument and look at the welfare benefits of trade in Britain in the 1760s and the 1850s. They find only a small impact of trade (with everyone else in the world, not just colonies) but a much bigger one in the 1850s. But their estimate for the 1850s is welfare losses of 25%-30%: large but if national income doubled then that leaves a 70% increase to explain.
So in summary, colonialism itself arguably harmed economic growth and trade, all trade, not just colonial trade, doesn't seem to have been big enough to explain much of British economic growth around the British Industrial Revolution.
Sources (partial as I have to dash off to some appointments)
"The Industrial Revolution: A Survey," a new essay, in Floud and McCloskey, eds., The Economic History of Britain, 1700-Present, 2nd ed., 1994.
"Keukentafel Economics and the History of British Imperialism," South African Economic History Review 21 (Sept 2006): 171-176.
Clark, O'Rourke and Taylor, 2014, The growing dependence of Britain on trade during the industrial revolution, Scandinavian Economic History Review.