Frankly they didn't need to. The Germans got everything they needed from Sweden. The most important thing was minerals, specifically iron ore which was critical to the German war effort. The major reason for invading Norway was to secure access to the port of Narvik so that Germany could receive iron ore all year long (the Baltic Sea freezes in the winter). The swedes also allowed the Germans to use their railways and Germans troops could pass through Sweden. Invading Sweden would just add another huge area to patrol and partisans to fight.

But, not to demonizes the swedes. They did shelter many Jews and war refugees. They also were a hotbed for resistance fighters. So they did do some good.

http://archive.adl.org/braun/dim_14_1_neutrality_europe.html#author

This article is well written and gives an overview of the neutral nations during the war. (It's a little hostile, so keep that in mind)