Hitler was not particularly fond of German speaking Swiss. He considered them race traitors with more loyalty to France than Germany. However, the Nazis did plan to invade eventually. After the fall of France, a plan was devised called "Operation Tannenbaum" which called for a total of 11 German and 15 Italian divisions (about half a million soldiers). Like in France, the Germans planned to invade one area first with a distraction force, with the intention to get the Swiss to send reinforcements to repel them, weakening their defenses along the rest of the border. The German main force would then smash though the weakened Swiss lines at another point and move behind the Swiss to cut them off. This is the classic Blitzkrieg doctrine that had brought them spectacular success.

The invasion never happened, mainly because the Germans considered Switzerland low priority and could be dealt with after their more powerful foes. Switzerland did not pose any offensive threat to Germany and was surrounded on all sides by the Axis for most of the war. After the invasion of Normandy the plan was shelved indefinitely.

In my opinion, not pulling the trigger on Operation Tannenbaum was smart the on Germans part. It had potential to be a bloody and difficult fiasco. The Alpine region of Switzerland is a natural fortress with only a few narrow invasion paths. Though the 1940 Wermacht was far superior to the Swiss Army, their geography is a huge force multiplier. The German Blitzkreig doctrine was developed for open countryside, not narrow mountain passes. The Germans were aware of this, and the terrain of the area Operation Tannnenbaum was designed to draw the Swiss Army into was favorable to the Germans. Personally, I don't think the Swiss would have fallen into the trap. The Swiss knew what advantages they held and that they couldn't beat the Germans in symmetric warfare. Getting drawn out into open ground ran directly counter to the Swiss strategy. They actually planned to abandon the area where the Germans wanted to fight them, and concentrate their forces in the Alps where they would bog down the Germans in a bloody war of attrition.

edit: clarification