I have read that Hitler had plans to invade Switzerland and Ireland. If Nazi Germany had won, it seems likely to me (edit: that's just my uneducated opinion) that they would have at least imposed a great influence upon the neutral countries, encouraging the political landscape to become more fascist. So why did neutral powers (excluding fascists) not take up arms to make this less likely?
Hello, I am very late, but I can at least partially answer for Switzerland.
Switzerland's neutrality is quite the myth in politics then and now, and entering a war would have had quite the political fallout. In a system based on consensus govenrment, this could tear the country apart, potentially weakening it.
Furthermore, Switzerland had a good business relationship with both Germany and the allies during WWII. The elite simply had no reason to enter the war, because it would have been both politically and financially very problematic.