How and why was the decision to dismantle the Holy Roman Empire made?

Short Answer: The French. Long Answer: These Goddamn French Are In My Holy Roman Empire Ruining My Shit.

Well, not entirely. Let’s get into the primary reasons why the Empire ceased to exist, according to scholarship and Peter H. Wilson in particular:

The Treaty of Basel made clear to the territorial Princes of the Empire that the Emperor could no longer protect them.
Mediatizations and annexations of free cities and ecclesiastical principalities following the Treaty of Campo Fiorno destroyed major pillars of the Imperial system.
There was a real danger in 1806 that Napoleon would have attempted to lay claim to the Imperial crown.

Before, though, a word on what was being dismantled.

Rehabilitating Voltaire’s Caricature

It is widely agreed among scholars that the traditional interpretation of the Holy Roman Empire, as a monstrous mess with no coherency after 1648, is mostly inaccurate. Motivated by the hostility of many enlightenment thinkers like Voltaire to the Empire, and nineteenth century nationalism, German historians of the Empire (in particular, though this was by no means just a German trend) played up its weaknesses and painted the picture of a decrepit, decaying anachronism.

This was not actually the case. Most of the Imperial institutions functioned reasonably well until the very last years of its life. The court system worked, the military circles were still effective or in the north had at least symbolic value (and in fact the Swabian and Bavarian circles became stronger in the latter half of the eighteenth century), and although large territorial princes and Prussia were no longer really as accountable to Imperial jurisdiction, they couldn’t just ignore it. Even Frederick II of Prussia, who never really understood the empire, resigned himself to its existence and tried to use it to further Prussia's aims later in his life.

When the French Revolution hit in 1789, most of the empire's members were still broadly for it. It gave privileges and existence to imperial institutions like the free cities, it gave protection to tiny estates, and it gave a shield of security to larger principalities against external aggression, as well as a source of prestige and the ability to share in potential spoils in the Imperial pie.

One of the mistakes that is often made in popular ideas of history is the assumption that old institutions cannot adapt. Just because the papacy or the Holy Roman Empire were old does not mean they were “decrepit” and had no ability to transform themselves. In fact, the very longevity of the Empire speaks to its robustness.

However, let’s make no mistake here; when the empire was faced with a major external crisis in the form of the French Revolution, it failed.

The Frenchmen are Coming!

The military failure of Prussia and Austria against the French during the War of the First Coalition, and France’s subsequent annexation of the principalities west of the Rhine revealed that the Empire, despite all its venerability and attempts during the war with France, simply could not reform its military fast enough to meet the French threat.

The Austrian defeat also revealed that the Emperor, traditionally the guardian of the member entities of the Empire, was unable to do his duty, in a dramatic way. The Treaty of Basel was a kind of concussive blow to confidence in the Empire; the entire left bank of the Rhine had been annexed and the Emperor could do nothing about it.

At the same time, concerning trends undermined the social contract between emperor and member entities. When Prussia withdrew from war with France, Austria acted in an aggressive way towards the remaining states in the war against France, and when the Treaty of Campo Fiorno was signed, the principle of compensation for lands lost to annexation by France was introduced.

In a desperate situation where more aggressive manhandling of imperial institutions by Austria was becoming normalized, the imperial institutions like the ecclesiastical estates found themselves victim of opportunism and demands for compensation by the larger territorial states. The Church Estates and all but six of the free cities were gobbled up, with massive losses.

The End Of Empire

Loss of confidence in Austria led to states acting independently. Württemberg seized Imperial Knight estates in 1803 against Austrian objections, and the total rout against the French had undermined any solidarity within the Empire. Austria and Prussia’s aggressive actions during war with France alienated the “Third Germany” represented principally by Bavaria, Württemberg, and Baden but also composed of hundreds of other states.

By 1806 Austria had little choice. With major defections either by choice or under coercion to France by the Third Germany, Francis II of Austria was in danger of losing his Crown to Napoleon by relinquishing it and having the electors appoint Napoleon. This is not speculation; Napoleon received an imperial title and actively tried to maneuver himself into a position where he could be legally elected. Complicated political games involving a hereditary title as “Emperor of Austria” had been conducted by Francis II since 1804 in response, and the loss at Austerlitz made things evermore imperative.

Claiming the title of Holy Roman Emperor would have been a disaster for Austria and a propaganda coup for Napoleon; he would have could wield imperial tools as he wished, and the empire still held some power in the minds of German rulers. On the 31st of July, 1806, even as Napoleon was moving to create the new German Confederation in the face of continuing Austrian obstinacy in conceding the Imperial Crown (there were hopes that if they conceded, they might get the Tirol back), Francis II abdicated and dissolved the empire.

Reactions to the empire, Wilson describes, were somewhat analogous to the death of a sickly old friend; you know that he's likely going to die, but when he does you still manage to be surprised. The empire had existed for so long and survived so many challenges that even when its preservation seemed impossible, it was still possible to imagine that by some unknown force it might have continued on against all odds until the end of time. As it was, it was destroyed by shocks it could not recover from.

Sources

I generally dislike using one author too much, but it’s very difficult to avoid Peter Wilson when it comes to the Holy Roman Empire. Dude’s everywhere; you can’t escape his work, and it’s excellent work.

Milton, Patrick. “Intervening Against Tyrannical Rule in the Holy Roman Empire during the Seventeeth and Eighteenth Centuries.” German History 33:1 (2015): 1-29.

Weis, Eberhard. “The Enlightenment and Absolutism in the Holy Roman Empire: Thoughts on Enlightened Absolutism in Germany.” The Journal of Modern History 58 Supplement (1986): 181-197.

Wilson, Peter H. “Bolstering the Prestige of the Habsburgs: The End of the Holy Roman Empire in 1806”. The International History Review 28:4 (2006): 709-736. (Main Source used here).

Wilson, Peter H. “Prussia’s Relations with the Holy Roman Empire, 1740-1786.” The Historical Journal 51:2 (2008): 337-371.

Wilson, Peter H. “Still a Monstrosity? Some Reflections on Early Modern German Statehood”. The Historical Journal 49:2 (2006): 565-576

Whaley, Joachim. Germany and the Holy Roman Empire: Volume 2, The Peace of Westphalia to the dissolution of the Reich 1648-1806. Oxford: Oxford University Press, 2012.