During the Early Middle Ages until about, say, the mid eleventh century, the Baltic was dominated by Scandinavians. By the Late Middle Ages, however, the Hanseatic League was the dominant power. How did this transition come about?
Well, from the Scandinavian POV, the mid-11th century is the end of the Viking Age and start of their Middle Ages. In the 10th century, Kaupang in Norway and then Birka in Sweden had been abandoned, the latter possibly raided by Baltic tribes. Varangian trading posts in the east went into decline as well.
Even though the Scandinavian countries never established actual feudal systems, the Christening of the countries still brought upheavals and changes. The Germans had already gotten over this. In fact, in ecclesiastical terms, Scandinavia was already "dominated" by Germany without knowing it, in the sense that it all had been made territory of the Archbishoporic of Bremen.
In the 12th century, what was to become the Northern Crusades start. Bishop Absalon of Denmark goes off burning Wendish (Baltic Slavic) temples on Rügen, Saint Erik of Sweden goes on a (likely apocryphal) crusade to Finland. The Swedes take Finland, putting them on the borders of Novgorod (eventually stabilizing around Karelia, not that different from where the Finnish-Russian border is now). But the Swedes are still weak on the home front; in 1187 the Swedish town of Sigtuna (in a sense a successor to Birka) is burned by Baltic raiders.
In the 1150s, Lübeck was founded, and began the ascent of the Hanseatic takeover of the Baltic trade. By the 13th century, you have Stockholm being founded (and soon to become Hansa-dominated), Visby becoming the Hansa's most important port in the Baltic.
So you have Nogorod now ruling the old trade routes, and Sweden and Denmark occupied more with gaining territory in the Baltic region. Neither of them have very strong governments, and there's no national strategy for trade, leaving the door quite open for the Hansa to move in. I think you could attribute much of the Hansa's expansion to the weak central government of Middle-Age Denmark and Sweden, and much of its decline to their consolidation of national and royal power in the early-mid 16th century. And from then on to the Napoleonic Wars, the great game of dominium maris baltici was played by Denmark and Sweden once again, the 17th century seeing Sweden dominate the Baltic to a greater extent than any county before or since.
I suppose one must bear in mind that the Viking Age trade, although it was impressive in its scale, was still pretty primitively organized. It was mostly individual expeditions to wood-hutted trading outposts, a far cry from the stone cities and corporate trading of the Hanseatic League.
