The Holy Roman Empire lasted for an incredibly long period of time and changed dramatically in that time. While I can't talk much about the eventual dissolution of the HRE or its founding, I will try to answer your question by focusing on the Empire in the period leading up to and including the Thirty Years War and its immediate aftermath. Unfortunately I've just moved and most of my reference materials are still in boxes, so this will be a more general response than I would like, since I'll be going off of memory. That being said, let's dive into it:

The common refrain about the HRE in this period is that the Peace of Westphalia and the Thirty Years War in general marked the creation of the modern 'nation-state'. The implication is that any kind of concept of the "Empire" as an overarching body was dead and only a formality, with the individual Imperial princes acting as independent states. However, throughout this period, the Empire had endured because being within the Empire was beneficial to them. Belonging to the Empire brought protection and power beyond that what a state could theoretically hold based on its size alone. Recognition as an Imperial prince meant that you had a voice in the Reichstag which in turn gave you prestige and power, at least partially due to being subject to an Emperor rather than a "mere" King. Moreover, by the time of the Thirty Years War, the HRE was not just an overarching concept consisting of nothing but a common loyalty to the Emperor. While it may not have been a written Constitution in the American mold, the Imperial constitution was a clear legal concept, and being an Imperial prince granted you certain rights and privileges that were protected by the constitution. Similarly, there were institutions designed to protect those privileges, and allow disputes to be settled through legal affairs rather than feud or military conflict. The Reichskammergericht and Reichshofrat (Imperial Chamber Court and Aulic Council respectively) provided an imperial level court structure to handle disputes between princes and other legal issues within the Empire. Likewise, the Reichstag regularly voted taxes to the Emperor for common defense, and the system of "Imperial Circles" existed to support a dedicated Imperial Army that would be supported by the princes in order to wage Reichskrieg. It is very important to understand that the Empire had a strong distinction between the Emperor as a person, and the Emperor as an institution. As an Imperial prince, even if you despised the man, you often had a strong respect for the institution. Indeed, German has a distinction in the words it uses to refer to the two (which often leads to confusion in English). Kaiserliche is used to refer to the Emperor as a person, while Reichs is used to refer to the institutions of the Empire, as distinct from the person of the Emperor.

Granted, the Thirty Years War--in many ways--was a result of the failure of these institutions, but even following the conclusion of the War at the Peace of Westphalia, the institutions did not cease to exist. Many of them continued to function well into the 18th centuries, up to the dissolution of the Empire itself. In the Nine Years War and simultaneous Great Turkish War, many of the troops facing off against the armies of Louis XIV were not Austrian soldiers, but rather Imperial soldiers provided and supported by the various princes. While it may not have resembled a state as we envision it from a modern perspective, the Holy Roman Empire did in fact have institutions within it that handled the affairs of state. While the Imperial princes did sometimes pursue their own foreign policy, wage war against the Emperor, and other such things, there was much more beneath the surface of the HRE than it might seen.

Moving away from the HRE a bit, there's a bit of an assumption in your post: that the situation in the HRE was somehow unique to it. In reality, the system you describe, of myriad territories nominally subservient to a King but often with their own governments pursuing their own policy was hardly unique in medieval or early modern Europe. Recall that Burgundy was nominally a vassal of the King of France, but was often at war with France. Similarly, while maps of Europe might portray "Spain" as a single polity, in reality even its Iberian holdings consisted of two separate "Crowns" (Aragon and Castille), each of which in turn contained multiple kingdoms and other sub-polities, each of which may have had their own governments and their own politics. The modern, centralized, bureaucratic state is definitely a relatively modern creation, and in the time of the HRE, a much greater degree of power was held at a much lower level than one would suspect. This is not a condition unique to the HRE, but one that could be fairly widely found throughout many European polities at the time. I could go on, but suffice it to say, many modern maps create the impression that the HRE was uniquely fractured, as they display the imperial marble cake in the heart of Europe, while in turn portraying "France", "Spain", and "England" as unified entities. Should you wish, you could easily split many of those states into their constituent parts, revealing their much closer similarity to the Empire than common knowledge would suggest.

I apologize for the poor organization of this post, but I hope you've found it informative. Please feel free to ask any follow up question you may have.