Most short answer: It has everything to do with the Pope.

First, until the Golden Bull of 1356, German electors (which changed sometimes dramatically one election to another) chose the King of the Germans, variously called Rex Germaniae under Ludwig II*, Rex Francorum* under the Ottonians, and eventually Rex Romanorum in the high middle ages in defiance of the Papacy. The King of the Germans would then be crowned by the Pope, assuming he agreed with the new leader, as the Imperator, (emperor) in German, Kaiser. You are right to notice that there were frequent interregna at least until Maximilian I (emperor from 1508 - 1519, King of the Germans from 1493 - 1519), after which only one occurred in 1740 with Maria Theresa and disputes among German electors over her father's Pragmatic Sanction and Habsburgs as leaders of the Reich.

Why did these happen? Most often because the Pope would not recognize the elected German king as emperor. Relations between German emperors and Pope's were extremely complex: who was superior to the other? What about land ownership? Investiture? Responsibilities as Pope or Emperor in regards to complaints from the others realm? If the Pope had reason to doubt the elected king, they would often withhold emperorship until they could secure promises, oaths, or benefits increasing their spiritual and secular power at the cost of the future emperor's. In addition, there were some cases where the Germans elected two kings! The King and anti-King would then have skirmishes and battles, and a real king would not be declared usually unless the Pope crowned one as emperor, one king defeated the other, or one simply lived longer than the other and thereby became the only elected king.

I can predict a possible further question! Wouldn't lacking an emperor for years destabilize the realm? In some ways, yes. But in almost all other cases: no.

It would be wrong to assume interregna were a prominent or visible symptom of the Holy Roman Empire as a weak, fractured, or unstable realm. If it were so, then periods of long and violent interregna should show fragmentation, and the splitting off of territories or creation of new realms yet this did not happen. Why didn't Italy, or Burgundy split away, like in the 26 year interregnum after the death of Heinrich III in 1056? Because after the early middle Ages, whoever was king of the Germans was also king of Italy and Burgundy. As explained by Peter Wilson:

Otto I’s coronation in 962 permanently associated the imperial prerogatives and status with the position of German king, making Germany the Empire’s premier kingdom. Italy slipped into second place after having been primarily associated with the imperial title between 840 and 924. Otto’s defeat of Berengar II ended the sequence of separate Italian kings. Henceforth, whoever was German king was also king of Italy, even without a separate coronation. Burgundy emerged from the Carolingian middle kingdom (Lotharingia) in 879 and maintained a distinct existence despite being considered subordinate to the Empire from the late tenth century. After 1032, Burgundy passed to the German king, who assumed authority directly, as in Italy. [1]

So the German king was already ruler of the lands comprising the Holy Roman Empire, even if he was not the emperor. It is also worth noting that the position of King of the Germans was extensive in the Empire, especially from the high middle ages when more precise terms of land-possession, leasing, ownership, and noble title rankings developed under the Staufers. Due to centuries of contracts with noble families and the various German kings, all princes and numerous secular counts and barons were immediate vassals of the king. This was paramount in securing power for the king, as now most secular lords were direct vassals, and not intermediary vassals (such as being Count of Ansbach, under the Duke of Oberbayern, under the King of the Germans, instead of under the King of the Germans directly).

Indeed, even without an emperor, the survival of the empire was paramount to all those participating in it. All those special contracts reducing feudal dues, or granting special exemptions would become void if one left the very realm (and more importantly, the lineage of people, i.e. kings, in the position to uphold the contracts) guaranteeing the validity of those contracts. Once again to quote from Wilson: “Despite the absence of a crowned emperor, the princes still saw themselves as part of the wider Empire rather than simply lords of what were, still, relatively small territories. They embraced the Staufers’ concept of collective honor imperii, because feudalization had heightened their sense of being stakeholders in the established order.” [2]

Source:

[1] Wilson, Peter H.. Heart of Europe (p. 182). Harvard University Press. Kindle Edition.

[2] Wilson, Peter H.. Heart of Europe (pp. 379-380). Harvard University Press. Kindle Edition.