There's a kind of "conventional wisdom" whereby people believe that the Nazis intended to conquer all of Europe or even the world, whereas the truth is actually that domination of Europe did not require conquest from the standpoint of the Nazi government, so the military was only deployed when military necessity required it. If you consider the countries in continent Europe that Germany never invaded and the ones that it did invade, you can find several categories.

Some countries were invaded because they were direct targets, such as Poland, France, the Soviet Union. Others were invaded because it was necessary (from the standpoint of Germany) to do so for a larger military goal; e.g., the Benelux countries were invaded to provide access to France; Denmark was invaded to provide access to Norway, which in turn was invaded to prevent a military intervention Norway by the U.K. Many countries invaded late in the war were ones that had been on Germany's side but were under Italian control, which was broken. Italy (obviously) is the big one, but the Balkan states and Greece were also in this category. Another category was former allies who changed sides or abandoned the Axis side: Hungary, Romania, and Bulgaria.

Regarding the countries that were never invaded, they were either sympathetic neutrals (Spain and Portugal), cobelligerents until too late in the war for Germany to effectively counteract them (Finland), or neutrals providing some benefit to Germany (Switzerland with banking). Sweden was in the third group, providing vital steel for the production of German tanks. There is every reason to believe that, if Sweden had posed some significant threat to Germany's ability to wage war, it would have been invaded too. However, it is equally likely that it would have been submitted to a very lenient form of occupation (for the "Aryan" population) like that in Denmark (which actually maintained its civilian government led by a social-democratic political party throughout the war) or Norway (a literal Quisling government -- the guy who led it was named Quisling, thus the term). Since Nordic Europeans were considered Aryan in very sense by the Nazis, it was deemed to subject them to harsh occupation unless the situation warranted it.

A very thorough treatment of Nazi occupation policy is provided in Mark Mazower's Hitler's Empire: How the Nazis Ruled Europe. A good recent work on Nazi military history is Nicholas Stargardt's The German War.

A few things I think might have contributed to this.

• Sweden didn't have as much strategic importance as Norway, particularly after Norway was taken. The long Norwegian coastline opposite Great Britain on the North Sea would give the Royal Navy something else to worry about and would force the British military to keep at least some token force available as a deterrent opposite Norway. Germany could use Norway to help control the Baltic, and as a base to try to prevent supplies from being shipped to the northern Soviet ports. Also, with Norway under German control and Finland a German ally, there weren't many reasons for Sweden to declare for the Allies, and fewer ways that the Allies could support them in that case. Sweden staying neutral was in everyone's best interests.
• Sweden made some concessions as a neutral that Germany was interested in. They sold some supplies to Germany during the war, and allowed the German military to use its road network to transport troops on occasion. When a few Polish submarines were damaged early in the war and made it to Swedish ports for repairs, the Swedes refused British requests that they be allowed to leave and sail for Britain. They allowed Germany access to telephone and telegraph lines through their territory that connected Norway and Germany.
• The Swedes had a pretty good relationship with the Finns, and they helped support them during the Winter War, although not in an official capacity. Germany may have been concerned that, if they squashed Sweden brutally, it could cause some issues with Finland. The Finns would never be a offensive threat, but on the defensive they'd bled Russia pretty effectively during the Winter War and could do the same against Germany. A "live and let live" policy was acceptable to all three countries.
• Sweden was a minor player in the Baltic region. In this area, it was always a game of influence between Germany and the Soviet Union, and Sweden would never be able to challenge that power dynamic. Joining the wrong side could be disastrous, but being "friendly but not friends" with both could pay some dividends.

Eventually, when things started turning against Germany, the Swedes were able to start turning down German requests and start subtly supporting the Allies - Germany couldn't spare the troops for a show of force against Sweden, so there weren't many consequences to these actions. Their overall course during the war can be kind of summarized by "cooperate with Germany when we have to, refuse Germany when we can."