In order to understand Germany in the 18th and 19th century, you have to understand the Holy Roman Empire (HRE). The HRE began with Charlemagne in the 9th century, and, by the late 18th century, had evolved into an elective monarchy exclusively dominated by the Austrian Habsburgs (with one 8 year exception between 1740 and 1748, but that's a side story). Most of the HRE was indeed German, but it also included Italians, Poles, Czechs, Wends, and so many more, so calling it a predecessor to Germany is inaccurate. The emperors regarded themselves as greater than any single nation (why be an emperor who limits himself to the Germans?), even if most of their subjects were indeed Germans. The title "King of Germany" did exist, but it was used as a way to designate an heir, not in the way that, say, the "King of France" or "King of England" were, and the King of Germany was understood to be the King of Italy and Burgundy as well. As such, claims by nationalistically minded historians that the HRE was a precursor to a German state are, at best, highly anachronistic, and not how the residents of the Empire would have seen it. That said, the presence of the HRE certainly did inhibit the creation of a German kingdom, since that space was already filled by an Austria that would be very angry at anyone claiming to be the king of their Germans.

All that said, until the French Revolution, there wasn't a very big call for a German state, even among Germans. Populist nationalism wasn't really a thing yet, all the imperial princes and electors derived legitimacy and prestige from serving an emperor instead of a mere king, and the emperor gained prestige, authority and money from the Empire. The rise of Prussia in the 18th century was motivated by their desire to protect and gain power, not by a desire to unite Germany (most famously, Frederick the Great was incredibly dismissive of German culture, and spoke French whenever possible). The small princes, free cities, and ecclesiastical bishoprics were protected from the predations of the larger princes by the emperor's guarantee, and the larger princes were protected from the predations of both each other and of the larger states of Europe, such as France, Denmark, and the PLC. None of the princes were strong enough to strike out on their own, nor did they wish to. Generally speaking, the Reichstag (where the princes and the Emperor would meet to make decisions for the Empire) was marked by compromise and cautiousness, and did a good job of avoiding aggravating anyone too much. Of course, the Austro-Prussian Rivalry and the Silesian Wars between 1740 and 1763 placed tremendous strain on this framework, but through it all, Prussia never threatened to leave the Empire, and nothing that I've seen suggests that anyone even considered it. There was too much inertia, tradition, and privilege wrapped up in the Empire for anyone to leave it behind.

So this brings us all the way up to 1806, when the HRE was dissolved by the Emperor during the Napoleonic Wars. Why wasn't Germany unified after the Empire dissolved, though? Surely the best time to forge something new was after most of those principalities had been subjugated and then liberated by the French, right?

Well, no. Napoleon certainly wasn't interested in a powerful, united Germany on his borders, and neither was anyone at the Congress of Vienna after Napoleon was defeated. Everyone was set on returning the balance to power to normal, and a unified Germany would throw that all out of wack. Austria was more concerned with her territories in Hungary and the Balkans, and was content with soft power in Germany proper. Prussia would have liked more land, but they hadn't contributed as much to the defeat of Napoleon as the others, and they were by far the smallest of the great powers at the conference, so whatever dreams they had of uniting Germany under their flag were just that, dreams. The smaller principalities wanted to regain their lost independence, and so Germany ended up as a mess of small states between Prussia and Austria. There were only ~40 states compared to the 300+ of the HRE, but still, not very unified, and there still wasn't a Germany on any map.

What they did have was the German Confederation. The German Confederation was originally an alliance between the German states, that eventually developed into an economic union, the Zollverein, in 1834. Surprisingly, even this was resisted by the smaller states. They greatly valued their independence, and didn't like the thought of being economically reliant on Prussia. By this time, German nationalism was a powerful force among German liberals, but the princes (or rather, by this time, kings) were generally very conservative. A true opportunity for a united Germany wouldn't come up until the Revolutions of 1848, since Germany was one of the hardest hit areas. During the Revolution, a parliament in Frankfurt formed, and offered the Prussians the German crown. The Prussians refused, mostly because the Frankfurt parliament didn't really have a crown to give. The Revolutions of 1848 are a tremendously complex and fascinating topic, one that I'm really not qualified to speak in depth on, but that's the basics of where it intersects with German unification.

Of course, as a Victoria 2 player, you're likely marginally familiar with the rest. A cascade of Prussian victories against Austria and France lead to them establishing the North German Confederation, and then the German Empire, finally giving European maps "Germany". Even here, though, not all were content with unification, particularly Bavaria, all the way up to WWI. The extent to which Germany was Prussian was always a source of tension between North and South Germany, and the divide lasts on to today.

In conclusion, throughout German history, there have always been significant forces pushing against German unification. The HRE established the balance between Prussia and Austria that the small German states would exploit to maintain their independence from the two great German powers. This balance proved to be remarkably stable, withstanding both Napoleon and the Revolutions of 1848, and it was only when that balance was sharply disrupted in 1865-70 that unification became possible.

Editing things to sound nicer. This is the first big comment I've written on /r/AskHistorians, and I've never appreciated how much goes into making them flow well!

To be a little glib here, one can respond that Germans didn't unify into one nation, because Austria was excluded from German Empire.

I think a deeper issue is that I am not sure that any of the states you are talking about had singular nation-states. France had a consistent set of borders for a fairly long time, but Peasants into Frenchmen makes a fairly convincing argument that it was not until the 19th century that the provinces were nationalized (e.g. where French/Parisian values were indoctrinated into the populace, where standard French was properly taught to the populace, etc.). Similarly, when you are talking of England, do you mean the United Kingdom, which has four nations (the English, Scots, Irish, and Welsh)? And the Russian Empire had numerous different nationalities inside of it that were never fully Russified (e.g. The Grand Duchy of Finland, but also Russian Turkestan and large Yiddish speaking Jewish populations along with Baltic Germans).

The issues that hampered German unification included different nationalisms. For example, you might be a Prussian nationalist who believed in dying for Prussia, but did not much care for the other German nations. Similarly, there was debate about who exactly was going to be in the German nation-state. Was it going to be a Greater German Solution (which would include Austria) or a Smaller German Solution (which would exclude Austria)? Who was in and who was out would determine who would rule the new German state and so different states had different interests in this. (E.g. If you were a Prussian, you might well like Austria out because that meant that Prussia almost certainly would dominate.) If Austria were included, would all of subjects of the Austrian Empire -- many of whom were not German -- be included or would Austria have to give up her empire (and really, why would Austria want to do that)? And, of course, who was in charge of the German nation? The revolutionaries of 1848 along were divided between moderates and radicals, but generally favored something between a republic (on the radical end) and a constitutional monarchy (on the moderate end), but the more conservative factions -- monarchs for example -- might well be uncomfortable with having their power so circumscribed. Incidentally, these faultlines are one of the reasons for the failures of 48.

So, to put that in brief, there was the question of what nationalism should be (Prussian, German, Reich-based), who should be included (Prussia + small states; Prussia + Austria + small states; how to deal with non-German subjects; whether territorial claims should be pressed regardless of ethnic claims), and what form the German nation-state would take (constitutional monarchy? Republic?), which combined with players who were uninterested in losing power (sovereigns), placed something of a strain on an attempt to unify Germany.

If you are instead asking why Germany failed to develop as a territorial state with centralized governance, or to put that another way, why the Holy Roman Empire failed to centralize as say France or the UK did, that's probably a bit harder to say, but I would think this has to do with the divided power in the Holy Roman Empire -- the various princes were joined in a loose confederacy and therefore the ability of power to be centralized in one person (the emperor) or one body (the diet), was somewhat limited. As to why it was a loose confederation, that would be because it never really had the goal of being a centralized state, because sovereignty was not centralized when it began (e.g. think about the King of England also being the Duke of Aquitaine), and because once the system was in place, it actually did, at times, help the states that were members of it.

Useful Sources:

The Long Nineteenth Century: A History of Germany, 1780-1918, by Blackbourn. The third chapter goes into a fair amount of detail about 1848. More generally the book discusses the formation of Germany.

The Oxford Handbook of German History Chapters 11 and 13 go into some detail about the formation of German nationalism.

The Holy Roman Empire, Reconsidered Collection of essays. Goes into a fair amount of detail on the Holy Roman Empire as symbolic order and one that subjects of the Holy Roman Empire would actively create (through ritualistic or legal means).

If you are interested more broadly in state formation, you could also read:

Charles Tilly's "War Making and State Making as Organized Crime" which tries to give a general theory for state formation. [This is freely available online, by the by!]

Charles Tilly's Coercion, Capital, and European States, AD 990–1992 Which might more directly answer your question [This is not easily available online!]

Finally, for nationalism more generally:

Benedict Anderson's Imagined Communities, Ernest Gellner's Nations and Nationalism, and Hobsbawm's Nations and Nationalism Since 1780: Programme, Myth, Reality

As a follow-up and more specific question, why did Austria not attempt to join with Prussia and dominate other smaller German states? How did the independent Austrian identity develop and why did they resist joining the German Confederation, given their shared language, similar cultures, and history of semi-unity in the Holy Roman Empire?