I want to stick to the facts -- what happened that we know of -- but I guess in the back of my mind is a "what if" fallacy: what if Napoleon just stopped all campaigns in 1810 and started consolidating the peace, reopening trade and commerce, etc. Could the French Empire have lasted?
It wasn't so much that they had immediate plans for war with Napoleon (although Portugal, Spain and Great Britain were actively engaged in war with France at the time), but that Napoleon had made himself and the French Empire into an obstacle to the balance of power within Europe. By 1810, France was Europe's preeminent state, in a position of either direct, nominal, or hegemonic control over pretty much all of Western and Central Europe. The other 'Great Powers' - Austria, Prussia, Russia, Great Britain - couldn't accept this in the long term.
Great Britain couldn't for two reasons: one, Napoleon had declared himself an enemy for life of Britain's, and his efforts to unite the continent directly threatened British interests and security. Austria and Prussia had been defeated by Napoleon and brought to terms - but their own national pride and Napoleon's insistence on the enforcement of the Continental System - which damaged their own trade and inherently meant bowing to the dictates of France - meant that it was an uneasy one. Russia was the great potential rival to France, Tsar Alexander hoping to have his own hegemony over eastern Europe - a dream that was obviously threatened by French power.
Further, peace in 1810 would require two things: one, reconciliation between Britain and France. Britain, as I have previously noted, was threatened by a dominant continental power, but also by French control of the Low Countries. Period sources tend to describe Antwerp as 'a pistol pointing at the heart of England', because it was a large port with easy access to the Thames estuary and thus London. Britain could not accept French control of Antwerp, or indeed anything but Dutch independence. The second thing it would require (related to the first) was the abandonment of the Continental System. Again, this was unacceptable to Napoleon; the Berlin Decrees were his chief method - indeed, his only real method - of attacking Britain, and their enforcement became the Empire's chief foreign-policy directive, underlying the invasion of Portugal and of Russia. Abandoning it would have required allowing other powers a degree of autonomy Napoleon seemed unwilling to give them.
Indeed, the greatest obstacle to a permanent peace towards the end of the Napoleonic Wars appears to be Napoleon himself. I'm talking here from a background of english-language historiography, but the consensus among historians is that Napoleon was unwilling to accept any peace but a dictated one - that is, he had to be in control of peace talks and enforce a peace on his own terms. To quote Chandler, "If there was to be any peace in Europe, it was to be pax romana". Any country that did not align itself with French interests was treated as an enemy, obliterating the possibility of a neutral, uncommitted position - which of course led countries that might otherwise have been neutral to take up arms against France, since, in their view, they would be viewed by Napoleon as enemies anyway.
So while the tipping-point for the French Empire was indeed the invasion of Russia in 1812, Napoleon's policies and attitudes as Emperor of the French were leading towards something along those lines in any case - and European powers were definitely considering the possibility of renewed war with France before that date.
Sources:
The Campaigns of Napoleon by David Chandler
Napoleon's Wars by Charles Esdaile