Stop 'fearing' invasion is a bit difficult to answer exactly, particularly sense it differs for both parties. One has to remember that British North America/Canada has always been smaller than the United States. I'll try to answer this question from both perspectives then.
For the United States, I think we can say with relative certainty that by the 1850s, fear of invasion by the UK woudl have been in the rear-view mirror for most Americans and the government. Lincoln's incredibly prescient Lyceum Address sums it up nicely: "-At what point shall we expect the approach of danger? By what means shall we fortify against it?-- Shall we expect some transatlantic military giant, to step the Ocean, and crush us at a blow? Never!--All the armies of Europe, Asia and Africa combined, with all the treasure of the earth (our own excepted) in their military chest; with a Buonaparte for a commander, could not by force, take a drink from the Ohio, or make a track on the Blue Ridge, in a trial of a thousand years." That was written in 1838.
However, I think if one wants a definitive conclusion besides just speeches, the Webster–Ashburton Treaty and the Oregon Treaty (1842 and 1846 respectively) both demonstrated that the two powers had more in common than they had in opposition to one another, and that even these very contentious issues could be peacefully resolved. However, we can probably go back even farther to 1815 and the Battle of New Orleans to really show where the fear was definitively put to bed for most Americans. Although the White House was burned in the war of 1812, the British were really unable to project their power at all inland more than a few miles at this point. Sure, the United States' coasts were vulnerable to the Royal Navy, but so was every other country.
For Canada, things were a bit different. The burning of York left a lasting memory in the minds of many in Upper Canada, while the Treaties mentioned previously did not look like a harmonious peace deal but rather an abandonment by the government in London. In the 1850s it was still very possible in the minds of the citizens of the various Canadian colonies (but again, most particularly Upper Canada) that the Americans would try to invade, or at the very least slowly eat away at the colonies by pressuring them to join the Union. It had, after all, happened to Texas, and Upper Canada, along with Columbia and Rupert's Land was becoming increasingly populated by Americans (who continue to make up a large part of the lineage Canadians). In this regard, one of the primary drivers for Confederation was the fear of American invasion and/or encroachment (which, again, those two treaties merely reinforced in the minds of Canadians). The expansion west and the heavy investment in linking up the two coasts via rail was heavily influenced by this as well (forcing the Canadian government to expend huge amounts of capital to make it happen as quickly as possible).
As for when it began to fade for Canadians, well, some might argue that the feeling of being overshadowed by the Americans never faded among us. However, in terms of actual risk of invasion, I think we can point to WW1 as the first real big solidifying of American/Canadian relations as they were tested in battle together. This marked the beginning of a shift for Canada from Britain to the US. The dedication of the Peace Arch (http://en.wikipedia.org/wiki/Peace_Arch) is sort of a final nail in that coffin.
That said, Canada did maintain a war plan in case of an American Invasion up until WW2, although that was likely just a theoretical exercise for bored officers.