You may be interested in an answer I wrote two months ago on a similar question. In short, firstly, Britain's colonies were probably a drain on wealth compared to a counter-factual of free-ish, peaceful-ish trade. Secondly, Britain's total trade, including with countries like France, was relatively unimportant to the British economy compared with domestic drivers.