Geographical (Hazardous terrain) issues alongside unnecessary use of resources, and Switzerland's neutral stance would, I assume be the main reasons...
If the war in Europe was more successful (in the long-term) for the Germans, would an invasion been likely?
You might find this previous response I wrote of interest. I'm happy to do my best with any followups you might have.