To understand why Sweden failed as a great power you have to understand why a small nation in northern Europe with minimal population ever became a great power in the first place because therein lies the answer. However, before we get into that, keep in mind that for all I'm about to tell you, Sweden is STILL just a small backwater naton in norther Europe with a minimal population. So how did Sweden ever become a great power? Well, let's break it down to a few key points:

1. Gustav I, better known as Gustav Vasa.
   Now we've all heard the story of how King Christian of Denmark offered the Swedish nobility amnesty only deliver the very opposite of amnesty. In fact he delivered what is known as the bloodbath of Stockholm and it is pretty much what it sounds like. See King Christian II was really fucking tired of his unruly Swedish subjects constantly revolting against his divine authority and he probably thought something along the lines of "Maybe if I cut their fucking heads off these pesky Swedish nobles will stop coming up with new ways to incite the population into open rebellion?". So chop chop, off with their heads and all that. Problem solved, right?
   No. Not quite. In fact, much like his amnesty was the opposite of what he said, his solution gave him the opposite of what he intended. Instead of stop the rebellions the cruelty displayed essentially united Sweden against Denmark. Previously the Swedish nobility had been divided, occasionally loyal to Denmark if it benefitted them, occasionally rebellious if that seemed like the best course of action. After the Stockholm bloodbath the idea of being loyal to Denmark was out of the question.
   So back to Gustav Vasa. He gathered a force of Swedish mountain men and miners, marched on Stockholm and lucked out because he was able to ally himself to the Hansa and, with their help, largely chased the Danes out of Denmark. This was possible because Denmark was temporarily involved in a civil war of their own. Turns out not even the Danes were all the fond of Christian. He was crowned King Gustav I and he turned out to be a pretty shrewd King for a guy who thought wearing MC Hammer's balloon pants in the 14th century was a good idea.
   Gustav I pulled off a series of radical reforms that effectively all served to strengthen the central authority in Sweden. Among them he went protestant, seized massive amounts of chruch property to pay the debt he had incurred to the Hansa for their help in kicking out the Danes. He also made monarchy hereditary rather than elective, dealt with a fair bit of internal strife, most notably the civil war known as the Dacke Feud.
2. Never go full feudal.
   Another key issue in creating a Swedish great power was not having to deal with a strong nobility. See, Swedish nobility was always comparatively weak and the peasantry strong because Sweden was never a feudal power and Swedish peasants, unlike their European counterparts, were general free men who owned their own land. This made them relatively rich (by peansant standards) and they frequently had enough land and time to engage in some kind of cash generating side business other than sustenance farming. Usually growing some kind of cash crop or engaging in some kind of craft. Why is that important? Because it greatly facilitated the switch from a barter economy to a cash based economy. And why is that important? Because taxes are a hell of a lot easier to gather in cash than in wheat and wool. So, a cash based economy (barter absolutely still existed though, it wasn't fully cash based) and a sherwd King in balloon pants with ambitions.
   Gustav I essentially conducted an inventory of all the land in Sweden. Estimated how much tax it could pay and created an effective tax collection. All of this meant that Sweden was waaaay ahead of the curve in terms of economic efficiency from the 16th century and onwards. All of this was possible because, back to point 2, Sweden never went full feudal. Gustav I could do all of this because he never really had to worry about powerful feudal lords whose lands existed almost as independent Kingdoms with the Kingdom. Neato.
3. Iron, silver, copper and war.
   Sweden had a lot of natural resources that were very important at the time. Iron, silver, copper, wood and again, a comparatively strong tax base. We also lucked out a fair bit with a well timed war against Russia in which we gained the port city of Reva (and all of Ingria). Reva was important because a lot of Russian trade passed through there and as such was subject to swedish tolls. More money for Sweden.
4. Just be really good at war.
   Sweden was good at war. Not because the average Swedish soldier is inherently more skilled than the average Dane, German or Frenchman but because we did war better. What does that mean? It means that Sweden understood that war is not about individual bravery. War is about organization and evolution and at that Sweden was very good. Charles IX changed the Swedish recruiting system to something akin to a conscription based system of part time soldiers. These soldiers trained for 2-3 months every year and this is important because at the time, this was often 2-3 months more than their continental counter-parts trained. Sweden was also good at learning from their mistakes so when the Polish hussars almost handed our asses to us in the Polish-Swedish war of 1600-1629 we learned from that and introduced effective cold steel cavalry of our own. We also introduced mobile smaller caliber artillery to infantry units, dastrically increasing the firepower of the Swedish battalions and introduced horse drawn artillery which allowed Swedish artillery to reposition much faster and adapt to changing battlefield situations more quickly.
   The Swedish infantry doctrine at the time, Gå-på (literally "walk on") was also extremely aggressive. This was possible because the Swedish infantry were actually trained to do this in a time when continental infantry tactics were based on battalions shooting at each other until one side broke and then in the best case scenario running them down with cavalry.Why did Sweden do this? Because we're a small backwards country with a tiny population in the ass end of nowhere. Fighting a war of attrition with nations larger than us by a factor of ten or more just wasn't an option. Sweden needed to win and win deciisively when they fought.
   And win they did. I think it's fair to say that for nearly 100 years most continental armies just weren't prepared to meet the combination of effective artillery, aggressive infantry tactics and even more aggressive cold steel cavalry. Sweden also produced some very skilled warrior kings who surrounded themselves with competent generals whom they could rely on.
5. Plunder and France
   How did Sweden pay for all of this? By being far ahead of the rest of Europe in terms of economic efficiency first of all. The tax income, tolls from the baltic trade and rich natural resources provided a strong economic base for Sweden. Added to this we went with the viking strategy of rape, pillage and plunder. We stole everything that wasn't nailed down from much of continental Europe for a long time and France was only too happy to provide Sweden with subsidies to keep the Habsburgs from getting ideas about uniting continental Europe under Habsburg rule.
   EDITS: I tried to fix the spelling and reddit went all Christian II on my formatting. Sorry.

Hey there,

Just to let you know, your question is fine, and we're letting it stand. However, you should be aware that questions framed as 'Why didn't X do Y' relatively often don't get an answer that meets our standards (in our experience as moderators). There are a few reasons for this. Firstly, it often can be difficult to prove the counterfactual: historians know much more about what happened than what might have happened. Secondly, 'why didn't X do Y' questions are sometimes phrased in an ahistorical way. It's worth remembering that people in the past couldn't see into the future, and they generally didn't have all the information we now have about their situations; things that look obvious now didn't necessarily look that way at the time.

If you end up not getting a response after a day or two, consider asking a new question focusing instead on why what happened did happen (rather than why what didn't happen didn't happen) - this kind of question is more likely to get a response in our experience. Hope this helps!