Sweden was, for example, declared independent during WW2, which led to over 50.000 Norwegian citizens and resistance fighters to evade the Nazis in Norway by traveling across the border. What made Nazi-Germany actually respect a country's wish to stay neutral even when so many were allowed to escape? Their plan was to conquer Europe so why not just take over a relatively easy target as Sweden, the same way as they did with Norway (who also wished to stay Neutral)
Perhaps a minor point but still worth expanding upon: As you acknowledge, the premise of the question hasn't always held. For example, in WW2, many neutral countries were invaded & occupied. Examples include:
| Neutral Country | Invaded/Occupied by | Date |
|---|---|---|
| Denmark | Germany | 1940-04 |
| Norway | Germany | 1940-04 |
| Belgium | Germany | 1940-05 |
| Netherlands | Germany | 1940-05 |
| Luxembourg | Germany | 1940-05 |
| Iceland | UK | 1940-05 |
| Lithuania | USSR | 1940-06 |
| Latvia | USSR | 1940-06 |
| Estonia | USSR | 1940-06 |
| Greece | Italy | 1940-10 |
| Yugoslavia | Germany | 1941-04 |
| Iran | UK/USSR | 1941-08 |
| Monaco | Italy | 1943-09 |
| San Marino | Germany | 1944-09 |
Note that while these countries were officially neutral, some of them provided aid to one side or another, which served as a justification for attack.