It seems to be that if a man like him was so hellbent on Pan-Germanism he'd think of all of the Germans, not just most. Even thought Switzerland was neutral, the Nazis could've just swarmed into the country, wouldn't you think?
More of course can be said, but you may find this answer to be of interest.