This thread by u/enygma9753 addressed this question recently.

The habitants (French settlers of Canada or "the Canadas" as it was sometimes called then) were Catholics, which the majority Protestant settlers of the Thirteen Colonies detested vociferously in a time of rampant sectarianism. New England opposed any accommodation to the "papist" faith and hoped that the 1759 British conquest of Quebec meant that, like the Acadians earlier, the French colonists would be displaced and disenfranchised, allowing for Protestant settlement.

In turn, the residents of Quebec -- while leery of their new British rulers -- had guaranteed protections under British law for their property, language and faith. They had very little trust in the New Englanders, who for the previous century and a half had attacked New France repeatedly and raided their frontier settlements. It was often a guerrilla war and all sides committed atrocities. There was no love lost between English and French colonists and, during the Revolution, the Quebecois were very skeptical of American pledges of liberty in exchange for joining the rebellion.

The Patriots didn't do their cause any favours when they briefly occupied Montreal with a heavy hand. Loyalists were harassed and Catholic clergy were suppressed. Although the Continental Army did recruit a few Canadian units, the vast majority of Quebec residents chose to remain neutral. They wouldn't take up arms to fight for Britain -- but they wouldn't rebel either. The Crown's accommodations largely muted attempts to stoke rebellion in the Quebec populace.

The colony of Nova Scotia had a large population of recent New England settlers who shared some of the grievances about taxation and their "rights as Englishmen" like their American cousins, but were less inclined to rebel (partly due to a religious revival sweeping the colony that had advocated for peace) and mostly preferred to be left alone to make a living. The local assembly was staunchy Loyalist and the port of Halifax was also the North American station of the Royal Navy. Frequent American privateering raids along the NS coast also turned public opinion against the Revolution.

George Washington was lukewarm about providing assistance to the Patriot cause in Nova Scotia, as he felt that there was insufficient local support for a revolt there. He wisely observed that any attempt by the Continental Army to liberate the colony would be likely regarded as a hostile invasion.