There used to live a baltic ethnic group there, who spoke a language similar to Latvian and Lithuanian. They were forced out by the teutonic knights, and the area was re-settled with Germans.

The name Prussia actually comes from the pre-german ethnic group, and is one the few world used in modern time from their language

These germans founded what used to be the Prussian empire, and they were dominant in the whole baltic area, forming the baltic-german minority. These events happend between 1200-1400 ( not quite sure about this one )

This german group was replaced by Russians and Poles after the second world war (1945-1960) .

** i know sources are wage, but gonns try to add some later. Im sitting on an airplane so its kinds hard.