The answer is pretty simple - most of the English-speaking Canadians at that time considered themselves British, and the French-speaking Canadians had a good deal with the British which allowed them to keep their language and religion.

Firstly, the involvement of settler citizens militias in the war of 1812 is kind of exaggerated. We like to talk about "Canadians kicking the American's asses", but most of the actual fighting was conducted by British regulars. Officers, amongst them General Brock, lamented that even though adult males were required to report for militia duty, many Canadian militia members didn't have functional weapons. That's not to say that they weren't important in particular battles, but they didn't make the bulk of the numbers. Indigenous allies of the British also participated in exchange for promises of land or security, like the famous Mohawk diplomat/general Thayendanegea, whose people's participation in the War of 1812 was rewarded with a land grant near modern-day Brantville, Ontario (his English name was Joseph Brant, thus the name).

Secondly, many of the British population near the American border, especially in New Brunswick and Southern Ontario, actually lived there on land granted by the British Crown because they or their relatives were Loyalists. These families had left the US because they didn't agree with the Revolution and wanted to stay with the Crown. So obviously they would never side with the very revolutionaries they'd fled! Even excepting the Loyalists, the vast majority of settlers in Upper Canada (modern Ontario) were British in origin, and the culture of pro-Imperial sentiment and Britishness remained for a long, long time thereafter.

Thirdly, French-speaking defenders had their own motivations, namely the preservation of their culture and language. Let's take the example of the Canadian Voltigeurs, a successful all-French-Canadian volunteer unit. The commanding officer of the Voltigeurs was a French-Canadian, Charles-Michel d'Irumberry de Salaberry, who was an officer in the British army. These men did not want to fight in an English-speaking unit but were very successful fighting together, with Indigenous allies, at the Battle of Chrysler's Farm, which is cited as a decisive victory in repelling the Americans from Lower Canada (modern Quebec). French Canadians had full legal rights to speak French, practice Catholicism, and keep French-style Civil Law within Quebec; rights that began with the Quebec Act of 1774. There was no guarantee that Americans would want to continue this arrangement, especially considering that they had cited the Quebec Act was one of the "Intolerable Acts", a list of British laws regarding British North America, cited by leaders of the American Revolution as reasons to revolt against the British government. They included the Quebec Act in the list because it made the Ohio River Valley part of the Quebec colony, but also because it established French and Catholicism as legal in Quebec - this was unacceptable to the Revolutionaries, who viewed Catholicism as a threat to their Protestant way of life. French Canadians, therefore, had no reason to want to join the United States; it would mean going from a place of legal protection for their culture to merging with an unknown, large, Protestant, Anglophone population that would likely be hostile to French and Catholicism. Siding with the States would be a terrible strategy for French Canadians to preserve their freedoms.