In the early middle ages, Norway was far more expansive than other parts of Scandinavia

Where'd that come from? The Danes took England! The Swedes went eastwards to a greater extent, making trips down the Volga, to the Black Sea and Constantinople, and as far east as Persia and Azerbadjan. The Rus' who founded Russia were likely from Sweden, and they later took Finland as well (in competition against the Rus'). You're talking as if vikings came from Norway exclusively.

Denmark is part of the European continent

I don't quite see why they'd be more powerful because Jutland is a peninsula rather than an island. Especially considering that historically, goods tended to move faster by boat and invading armies much faster over land. Last time Sweden proper was invaded (1809) - and there were relatively few of those considering the number of wars Sweden was in - Russian troops marched over the ice of Kvarken, with large casualties.

Anyway, Sweden has a larger population that's about the same or slightly higher than Denmark and Norway combined, both now as in 1900 as in 1810 and prior to that Sweden consisted of Finland as well all the way back to circa 1200. Why wouldn't the country with the largest population and territory be the most powerful?

Politics and disease took Norway out. The black plague hit Bergen, Norway in 1349. It killed an estimated 20% of the European population, but in Norway it killed 50%. Large swats of the country were left uninhabited. And regional politics played into it, leaving Norway without a king. Sweden and Denmark(also ruling Norway) for long periodes were close to equals, but Denmark had persistent border problems with German states, that Sweden was spared. The final blow came as Denmark supported the losing side during the Napoleonic wars, and had to hand Norway to Sweden as part of the peace settlement in Kiel 1814.