In the 18th century Austria was a badass, and still was by the end. But by the end of the 19th they were a second rate power with tons and tons of internal issues. What caused this?
One of the most important themes of European political history in the 19th century was the emergence of nationalism and its increasing role as the sole source of political legitimacy. The Austrian Empire was fundamentally an anti-nationalist polity - it consisted of several nations and peoples united by a single sovereign. As nationalism grew and grew over the course of the century, it chipped away at the Habsburgs's legitimacy, and thus its capacity to run an effective government.
The House of Habsburg spent the late medieval and early modern era compiling as many royal and noble titles as they could manage, controlling Austria, Hungary, the Netherlands, Bohemia, and a ton of others. Until 1804, the only thing that united these disparate places was the fact that their king happened to be the same person (a personal union). In 1804 amid the Napoleonic Wars, they formally tied all these places together and created the Austrian Empire. But fundamentally, the Habsburg Monarchy was the glue that held the Empire together, not any shared identity.
Following the enlightenment and French Revolution, nationalism - the idea that a nation or people (i.e. the French or the Germans) should be the basis of a political entity - became an increasingly dominant political force in Europe. The Austrians, under the political leadership of Metternich, did everything they could to stop nationalism, since their state was comprised of dozens of nationalities that now thought they should each rule themselves. Nationalism posed an existential threat to the Empire. It ignited a series of revolutions in 1848, and Austria narrowly avoided losing Italy, Hungary, Poland, Romania, and just about every other constituent part of the empire, even its German heartland (Austria proper). The revolutions failed, but at this point it was becoming clear that nationalism wasn't going anywhere.
But the point of no return came in the 1866. German nationalists for years had debated what a "German nation state" would look like - namely, if it would be dominated by Austria or Prussia, the two principal German powers. The Austrian solution - "Großdeutschland" (Greater Germany) - would have all German speakers in one nation, with Austria/the Habsburgs dominating politically. But this left open the question of what would happen with all of Austria's non-German imperial possessions - would they remain under Habsburg/German rule, or would something else happen to them? The Prussians favored a solution called "Kleindeutschland" (Little Germany), where only the non-Habsburg German states would join Germany, leaving out Austria proper and Bohemia. This would make Germany 100% German, but it would not contain 100% of all German people - the Austrians and Bohemians would be left out.
This question was decisively answered by the Austro-Prussian War in 1866, where Prussia defeated Austria and removed any influence the latter had on German polities and German nationalism. It paved the way for Prussia to create the German Empire in 1871, in accordance with the Kleindeutschland solution.
At the same time as the Austro-Prussian War, the Third Italian War of Independence tore away Austria's remaining imperial possessions in Italy. It looked like the Austrian Empire was going to splinter under the pressures of nationalism. In order to prevent further collapse, the empire reorganized into Austria-Hungary, after which it was very complicated set of independent polities, again united solely by the Habsburg monarchy. In the industrial era, when countries like France, Germany, and the UK were centralizing and building sophisticated modern economies and militaries, the de-structured and fractious nature of Austria-Hungary was a significant disadvantage, and by the time World War 1 came around, it simply couldn't compete with the Great Powers economically or militarily.