I wrote an answer previously on the subject of Sweden that I will post here.

Frankly they didn't need to. The Germans got everything they needed from Sweden. The most important thing was minerals, specifically iron ore which was critical to the German war effort. The major reason for invading Norway was to secure access to the port of Narvik so that Germany could receive iron ore all year long (the Baltic Sea freezes in the winter). The swedes also allowed the Germans to use their railways and Germans troops could pass through Sweden. Invading Sweden would just add another huge area to patrol and partisans to fight. But, not to demonizes the swedes. They did shelter many Jews and war refugees. They also were a hotbed for resistance fighters. So they did do some good.

http://archive.adl.org/braun/dim_14_1_neutrality_europe.html#author[1] This article is well written and gives an overview of the neutral nations during the war. (It's a little hostile, so keep that in mind)

As for Switzerland, the Germans considered it and even went as far as planning operation Tannebaum, an invasion of Switzerland. Invasion routes were planned and terrain was studied but Hitler never gave the operation a go ahead. Hitler felt that Switzerland could be erased once the Soviet Union and Britain were destroyed, but since that didn't happen Switzerland survived.

Source:

A World At Arms by Gerhard Weinberg

They would have liked to take Switzerland. It was planned. However if you look at the history of Switzerland, they are very, very good at defending themselves, mainly due to the terrain. With fortifications etc. they managed to deter the invasion, getting the message accross that it would be too costly.

As for their morale, which was traditionally high - check out Guisan. Basically the Swiss governent said it has no authority to order a national surrender, so everybody should fight to the death. That is fairly scary and a good deterrent.