Pan-German sentiment (calls for a unified German people under one nation, Greater Germany) had been increasingly pronounced throughout the 19th century. This fomenting idea was one of the factors contributing to the 1848 revolutions that swept through the many German states. Although ultimately the revolution was a failure, this idea continued to simmer and grow in both popular sentiment and various intellectual circles.
Meanwhile the state of Prussia itself was rapidly emerging as one of the Great Powers. Its geographical location in the Rhineland, access to abundant waterways and fertile plains allowed it to industrialise rapidly, compared to the more Alpine Austrian Empire. As a result Prussia could successfully form an economic and social hegemony over the other smaller German states. (the earlier German confederation was formed as a trade alliance actually, which Prussia began to grow more and more predominant over)
Prussia was also a militarily powerful country. Sandwiched between the French Republic to the West and the Russian Empire to the East, the German states could rely on an alliance with Prussia in the event of war with either. Frederick the Great and his legacy was instrumental in transforming Prussia into one of the great military powers in Europe after all. As it was oft quoted "other states have an army, but Prussia was an army with a state".
The Austrian Empire consisted of numerous other diverse ethnicities, like Croats, Czechs, Slovaks, Hungarians and others. This did not sit well with the Pan-Germanists who wanted a Greater Germany, consisted mainly of Germans. Prussia on the other hand WAS predominantly German (save for the significant Polish populations in the Eastern regions after the 1795 partitions of Poland)
Not only that but the Austrian Empire's strategic defeat in the Austro-Prussian war, it was clear who was the pre-eminent power among the German states. Whilst some of the states (like Saxony) actually fought on the side of Austria and against Prussia the outcome of the war made it certain who was the winning horse. The Austrian armies were made up of numerous ethnic minorities speaking various languages and it made military coordination exceedingly difficult. Not only that Austria was overall not as well coordinated and had a reputation of being technologically inferior.
But the largest factor that led the southern German states to join the North German confederation was actually the Franco Prussian war. The southern German states have a long historic enmity with the French, and when the Franco-Prussian war broke out they joined the Prussians and successfully won the French in various battles.
Bismarck had always been a shrewd negotiator and he succeeded in uniting the German states in the war effort and convincing the South Germans to put aside their differences and support unification and solidarity. Events culminated in the unification of the Southern German states with the North German confederation and most famously of all, the declaration of the German Empire in the Hall of Mirrors in Versailles Palace in January 1871.