Well, for starters Hitler didn't want to use precious resources and manpower on Scandinavia as a whole as he eventually ended up fighting a two front war. Sweden also remained neutral, even not allowing German planes to use Swedish airspace during other invasions of Scandinavian countries.

Germany had no good reason to invade Sweden.

They invaded Norway because Norwgian ports gave them good control over the North Sea, because they got a lot of iron from Norway and were afraid that the Allies would invade first and cut off their supply, and because Quisling convinced Hitler that Norway would be an easy victory.

Denmark , again, made for a good German naval base, and bordered Germany, so the Germans were worried that if the Allies took Denmark, they'd be able to launch direct attacks against Germany.

The Germans got a lot of iron from Sweden too , but by owning Denmark and Norway , there was no danger that the Allies could get enough troops to Sweden to invade and block German supply of the material. So ultimately, occupying Sweden just wasn't worth the hassle.