The difference in population has more to do with their colonial history. Canada was a French colony up until the 1750s, and at that time, Catholicism was the official religion of France and the only recognized religion in Canada. Contrast that with British North America, who were allowing non-Anglicans to set up religious settlements in the New World.
As such, the settlers in Canada in its first couple centuries weren't really escaping anything. They were coming to Canada to make money in the fur trade, then return to France (or elsewhere in Europe). There certainly were permanent settlers in Canada pretty much from the start, but not to the same degree as there were in the future U.S.
After the French and Indian War (a.k.a. The Seven Years War) ended in the 1760s, France handed control of Canada to Great Britain. As far as your question is concerned, nothing much changed in Canada at first, until the aftermath of the American Revolution. Between 1780 and 1800, 50,000 ex-Americans moved from the new U.S. to Canada (mostly Nova Scotia, Newfoundland, and the part of Quebec that became Ontario years later) to claim free land on offer to Loyalists (see: U.E.L.) This was the biggest English-speaking immigration the country had seen and really forms the base of the non-French Canadian population.
The Canadian provinces were united in 1867 where modern day Canada began. Before that, the British limited the officially recognized religions in Canada to be Catholicism, Episcopalianism/Anglican, Presbyterianism, and, IIRC, Quakers, and Baptists. As such, the appeal of immigrating to Canada was limited to Europeans of particular backgrounds.
Contrast that to the first half of 19th Century U.S. immigration, with its First Amendment protections, where many Germans (Calvinists), as well as Irish and English continued to come to avoid religious and war issues back home.
TL;DR: The difference in population is less to do with weather, and more to do with difference in the immigration motivations of colonists in each country. Canadian colonists tended more to come to make money, then leave, where American colonists tended to buy a farm and raise a family. Canada was also much more religiously closed until the mid-1800s. By the time of Canadian Confederation in 1867, the U.S. had attracted substantially more immigrants, and the population differences persist to this day.
hi! you may be interested in some previous responses