I am curious because I remember reading there were instances where there was a push to annex all of Mexico, going even as far south as Panama after the Mexican American War. Likewise I have read that during the civil war, many prominent southern politicians sought to integrate parts of North America, the Caribbean (Cuba specifically) and again Mexico as slave states to counter the expansion of the non-slave states.
I remember reading the only serious attempt to exert direct control over Canada was during the war of 1812.
It seems interesting to me given the relative power of the U.S. and the concepts being pushed during it's younger years as a country (like manifest destiny) that it did not make a serious move to control more territory.
Remember that the US only really because considered much more than a regional power in the 20th century (generally accepted to having been recognized as such in the 1898 Treaty of Paris.) Prior to that, the idea of sparking armed conflicts with rival nations was not only likely to be ridiculously one-sided (as you mentioned with the War of 1812... having your capital burned ( photo ) to the ground is a great way to quell much thought about military adventurism for a while), but just not particularly high on the docket of US national interests. The last thing the US wanted was another war with Britain by poking Canada, Spain by flicking Mexico or the Caribbean, or heaven forbid their closest ally France by invading its territories ( map ca. 1803-1809 ). Fortunately, Napoleon was in a real cash bind, and sold the Louisiana Territories in what amounted to a fire sale (>$0.04/acre at the time, which is ~$0.42/acre today).
Moreover, there was no reason to go picking fights with the colonies of European powers, because the US already had all the territory it could ever want or need just waiting for them to the West. Of course, there were people already living there, but clearing them out was the business of the day. Manifest Destiny and it's tragic consequences are about as close to what you're describing as it got. Though the Mexican-American War in the late 1840's is sometimes said to have had the potential to have annexed all of Mexico, the North/South tensions were such that it would have been a really tough sell to get the Whig elements of the Northern and Southern states [thanks /u/BRBaraka] on board with adding territory south of 39°43′ N that might lead to a slave power ( Giddings, Joshua Reed, Speeches in Congress [1841–1852], 1853, p.17, full text), while Southern elements feared that Catholic Mexico would remain hostile to the institution of slavery and pen the Old South in (here's President James K. Polk's third annual address on the subject: link.) Moreover, Southern opponents to Mexican annexation voiced the argument that they wish to build the South through industry, not through further territorial expansion (and thus continued reliance on King Cotton agriculture that kept the Southern states lagging well behing their Northern counterparts). That tension, of course, erupted into the Civil War of the 1860's, and it's aftermath and reconstruction phase ate up much of the rest of the 19th century.
Couple with all of the realities on-the-ground, the American psyche didn't see itself as conquerors or invaders* (* of other "civilized" [read:European] nation-states). The national mindframe - however much it didn't mesh with the reality of westward expansion - virtually precluded the idea of forcibly annexing territory that had already been claimed by another power - unless that territory asked to be included (Texas being the great example). That, of course, would change with Teddy Roosevelt and the Jingos getting all hyped up on the idea of an American global empire... but that was a pretty short phase that lost much of its luster following the debacle in the Philippines from 1899-1902. By that point, the Age of Empire was effectively over.
Edit: changed the 2nd pp to make it not sound like I was trying to put the Mexican-American War after the Civil War!
In the mid-19th century there were a number of private American citizens who basically decided to launch wars of conquest in Central America or the Caribbean of their own accord. They became known as "filibusters".
William Walker was probably the most successful of the filibusters, he succeeded in controlling part of Nicaragua and declaring himself President of that nation.
Most of the filibusters were from the South and definitely used the idea of forming new slaves states as a means of convincing Southerners to support them financially, politically, or militarily. One could argue whether the filibusters themselves were genuinely interested in expanding slave territory or if they just used this to drum up support from Southern political and business interests. (Walker was a really strange guy, he was a doctor AND a lawyer by his early 20's.) Again, this is the mid 19th century and the political balance between free states and slaves states was front and center in American politics, so it was an effective way of appealing to people.
But the sharp political divide also made it so that the American government as a whole wasn't willing or able to support the filibusters. So in William Walker's case, where he had a degree of success and was attempting to consolidate power for the inevitable counter-attack, appealing to the US government for military aid wasn't really going to work. President Pierce even recognized Walker's government as legitimate, because he had essentially allied himself with a political faction in Nicaragua that gave him legitimacy when he took control. Southerners in the US supported Walker financially and politically, but he didn't get any direct military support, not that the US had a very impressive standing military at the time anyhow. You also had the fact that there were some significant British colonial interests in the same region that the US obviously was not looking to upset. (After Walker's regime failed he tried to drum up support for a second try, was arrested by the British in Honduras and turned over to the Hondurans, who had helped expel him from Nicaragua, and was executed.)
So to address your question, in the case of the filibusters you had individuals who were basically looking to conquer whatever looked good for the taking. Using the slave state issue was a convenient way to gain support for their endeavors, but using this issue also assured there would never be bipartisan support for what they were doing.