The Baltic states has at least since medieval times contained German speaking populations as well as other minorities. Estonia, especially, had a Swedish speaking minority predating Estonia being incorporated into the Swedish state in the 1500s.

The Baltic-Germans were in part a remainder of the Teutonic order's presence and mercantile activities of the Hanseatic League. Quite a few seem to have consisted of nobility and the Swedish Empire made good use of the Baltic-German aristocracy, many such families transitioned to being Swedish nobility and their descendents are found in Sweden and Finland to this day. Not that they always had it their way. When the Swedish state tried to abolish serfdom in it's Baltic provinces some decided Russia was a better master and helped ensure the provinces were conquered in the 1700s. I should add any Swedish nobility who gained estates in the Baltic provinces had no qualms about taking advantage of their greater power over tenants and also resisted the incorporation of the Baltic provinces into normal Swedish counties.

By the 1900s the situation is mildly put complicated, the Baltic-Germans had a somewhat fractured relationship with the new Baltic nation-states after WW1 since they didn't exactly fit into the one nation one people model that sort was how the political order was shaking out.

In short, most of the evacuated people would have been of natives of Baltic-German extraction or civilians who had followed the German army and civil administration into the Baltic regions. The Königsberg area, East Prussia, today's Kaliningrad exclave, was a majority German area and would have been filled with Germans up until the end of the war.

Incidentally, many Estonians who were Swedish speakers also wanted to be evacuated, and many managed to travel to Sweden. An unfortunate large number of those were returned due to Soviet demands and would suffer under Stalin. A shameful stain on Sweden and it's appeasement policies IMO.