There are a few reasons for this.

​

Many Northern State politicians believed that Canada would one day join the Union on its own (After all there was a clause in the Articles of Confederation for Quebec to join the confederation if it wanted to). There would be no need for war. And war with the British Empire would hinder the northern states ability to trade with Europe. The Northern states were much more industrial and depend more on mercantilism and trade than the early agricultural economy of the south. The British Empire had by far the largest navy in the world so foreign trade would be made nearly impossible in case of war between the two nations.

Southern state politicians did not want Canada to join the union because it would only add free states. Just as many northern politicians were skeptical of adding Texas to the union because conversely it would be added as a slave state. Antebellum America was growing increasing concerned about balancing slave states and free states after the Missouri Compromise in 1820. Keeping the balance seem essential to preserving the Union.

The Canadians and British didn't want a war because it would have been devastating. The Canadians would not likely to be able defend themselves from an American invasion. The American forces that invaded Canada in the War of 1812 were lead by extremely incompetent generals (this should not be a controversial remark). Also, during the War of 1812 the American militia that was meant to help conquer Canada often refused to enter Canada. If a more competent and dedicated US wanted to conquer Canada in the War of 1812 may have had a different outcome.

While there were some disagreements (mostly border disputes and for influence in Central America) between the US and the British Empire and both sides did not trust each other for much of the 18 and 19th century. Both sides generally agreed war would benefit no one. While diplomacy was often contentious, cooler heads prevailed war was to be avoided.