I would assume that it would be because of there being so much rugged terrain and not much to gain from it, but then again Switzerland is a sort of political capital of Europe, so you'd think it would be a good blow to deal.
Not to stifle further discussion but /u/Georgy_K_Zhukov has discussed this at length here
The Nazis considered the invasion of Switzerland, called Operation Tannenbaum, until late in the war. It is in the nature of a military to plan for improbable contingencies - the Wehrmacht also developed plans for invasions or occupations of Iberia (Operation Felix), Ireland (Case Green), Britain (Operation Sea Lion), and Iceland (Operation Icarus) - so this doesn't mean much in isolation, but Hitler does seem to have sincerely intended to occupy Switzerland and incorporate at least a portion of it into the Reich. This makes sense when you consider his revanchist and nationalist attitudes toward other regions with large German populations - the Sudetenland, the Danzig corridor, Alsace-Lorraine, Austria, Flanders. A significant portion of Switzerland was ethnically and culturally German, and Germans belonged within the Reich.
There were, however, good reasons that this never happened. First, there were other, more pressing strategic concerns (not the least of which, before the summer of 1940, were the facts that Italy was not yet a German co-belligerent and France was unconquered). Second, Switzerland maintained a large, mobilized military during the war, with articulated defense plans; their so-called "National Redoubt" consisted of fortified positions guarding Switzerland's alpine highlands. This was perhaps not enough to prevent a German-Italian conquest of the country, but it was certainly enough to render it more trouble than it was worth.
Finally, the Swiss played a deft diplomatic game throughout the conflict, offering both sides opportunities that vouchsafed Swiss neutrality - particularly the German side. Swiss banks held German money (and money stolen by Germans from Jews); Germany did international business using Swiss Francs purchased with gold; Switzerland served as a go-between to facilitate contact between Germany and the Allies; Switzerland was fertile ground for intelligence-gathering; Switzerland offered Germany limited rights of rail transit through the country; and Switzerland carried on a booming trade with Germany throughout the war. So you might argue that Switzerland avoided a German invasion not because Germany couldn't conquer them, but because their concessions and accommodations rendered a German conquest superfluous.