Americans actually had an enduring assumption that Canada would eventually fall into their hands - as late as the 1890s, there was talk about Canada getting absorbed into the States. They were wrong each time.
The main factor which Americans might have thought would work in their favour was the relatively undemocratic government in Canada. Canada didn't yet have "responsible government" - that is, a government that can be recalled by the people's representatives. That only happened in the 1840s - 50s (different times for different provinces, Nova Scotia first in 1848). Governors, appointed by the Crown, had much more power and selected cabinets to run their province, consulting with the elected officials but they could not be taken out of office by the elected officials. There was a strong argument that that was an unrepresentative and unfair system.

Why didn't these factors actually motivate Upper Canadians? Around the time of the War of 1812 a significant proportion of the population were Loyalists and descendants - they had chosen to leave the US when it revolted, why would they revolt against Britain now? 7,000 Loyalists settled in Ontario immediately during the Revolution, and then more arrived later under the title of Loyalist. (Unfortunately, the census data available via the Queen's library doesn't have ethnic or place-of-birth breakdowns until the 1840s.) Out of a total population of about 76,000, this was a pretty significant group, and more significantly for the odds of an American invasion, they mostly settled along the Saint Lawrence, which is the exact area the American forces would be invading. And as you note, most of the European population had settled as farmers and received land grants from the British government, free or subsidized land, which would also tip them in the favour of the British government. Upper Canadians didn't become activist about Responsible Government until the late 1830s, for example, in the Upper Canada Rebellion of 1838.

Additionally, Americans assumed that French-Canadians in Lower Canada would be willing to revolt on a cultural basis. The memory of the British conquest of Quebec in 1758 was there and Americans assumed it would be a great motivation to want to fight the British, but ultimately the United States represented a much larger Anglophone/Protestant population that would further outnumber the French and Catholic voice in government and culture. There was also a large contingent of anti-Catholicism in the United States, which would be a threat to the Catholic religion in French Canada. Essentially, if Quebec were to become part of the US, its inhabitants were likely to be culturally assimilated. The Quebec Act, toleration of Catholicism, and the provincial legislature of Quebec were good enough for the average Quebecer to be satisfied, and they could preserve their culture better by keeping Catholic churches running and passing their language on to their kids than by trying to back the Americans.
Demographic is from: https://opentextbc.ca/postconfederation/chapter/1-2-historical-demography-of-canada-1608-1921/