I think /u/Witte49 provided a good answer, but I will try to add some details and built.
I can also point to my earlier post here where I touch on the subject.
The German navy was concerned about being locked into the North Sea by the British blockade and was pressuring Hitler to invade Norway to secure bases. The concern about the Swedish iron ore (about 10 million tons yearly, or slighly under half of the German need) that was transported from the mines around Kiruna to Luleå (in summer) or Narvik (in winter) along the ore railroad to be shipped to Germany (and other importers, Britain imported some 4 million tons yearly at times) was also important to the Germans.
The Allies had in February-March 1940 declared that they wanted to aid Finland by sending an expeditionary force through Narvik, Kiruna and Luleå to aid Finland, and planned to land and march to the Swedish iron ore mines and ports even without Swedish transit agreement (which was denied on the 2nd of March 1940).
The Germans were aware of the British and French preparations to occupy Norway and the Altmark incident on the 16th of February 1940, where a British destroyer siezed a German merchant vessel (secretly carrying British sailors made prisoners of war by the Graf Spee raids in the Southern Atlantic) in Norwegian waters, with a Norwegian destroyer present but doing nothing convinced the Germans that the Norwegians would not defend their neutrality against the British, so they decided to attack Norway before the British could establish a secure foothold.
Denmark was invaded as a stepping stone towards Norway, as the Germans needed the airfield at Aalborg in northern Denmark to support their forces in southern Norway.
With Norway in German hands, the Allies could not invade Sweden, so no plans were made for invading Sweden at the same time as Norway and Denmark. Besides, the German navy and merchant marine were extremely stretched as it was with the daring and adacious attack on Norway (which counted massively on luck and the element of surprise) - they simply did not have the capacity to invade Sweden at the same time. Considering the weak strategic position of Denmark (flat land with a border directly with Germany) and the weakness of the Norwegian armed forces, only small German land forces were required for the occupation, which meant that little was taken from the upcoming invasion of France and the Low Countries. While the Swedish armed forces were weak compared to the Germans, they were still twice as strong as the Danish and Norwegian ones together.
Invading Sweden thus required more resources and troops than the Germans were confortable taking from other fronts September 1939-August 1940 and June 1941-May1945. During the August 1940 to June 1941 window, Sweden did give the Germans almost anything they asked for, and the strategic position of Sweden was weak, as it was dependend on German coal and German agreement for a certain numbers of ships to pass both the British and German blockade in the North Sea to secure a food supply for its population.
To show the difference between the Norwegian and Swedish armed forces in April 1940:
| Category | Norway | Sweden |
|---|---|---|
| Army size | 120 000 | 400 000 |
| Conscription time | 85 days | 240 days |
| Heavy artillery | 0 | 96 |
| Medium artillery | 46 | 280 |
| Light artillery | 178 | 364 |
| Tanks | 1 | 78 |
| Armoured cars | 2 | 54 |
| Heavy AA guns | 36 | ~150 |
| Medium AA guns | 18 | ~400 |
| Light AA guns | 0 | 56 |
| AT guns | 0 | 444 |
| AT rifles | 0 | 450 |
| Fighters | 12 | 74 |
| Bombers | 4 | 96 |
| Recon planes | 56 | 95 |
| Torpedo bombers | 6 | 12 |
| Armoured ships | 2 | 7 |
| Cruisers | 0 | 2 |
| Destroyers | 7 | 15 |
| Submarines | 9 | 17 |
To sum it up:
The Swedish armed forces were strong enough during most of the war that the Germans would be required to take more forces than they were comfortable with from a more important front to invade Sweden.
During the period with the greatest disparity between available German forces and the Swedish armed forces, Sweden was very compliant with German demands.
Sweden was "protected" by a ring of states that meant that in order to invade it, they had to be invaded first. This means that the fighting for the Swedish iron ore happened between the Allies and Germans in Norway rather than Sweden.