The original inhabitants of Prussia were not Slavic to begin with, but rather Baltic people - somewhat related to Lithuanians and the neighboring Yotvingians who lived in what is modern day North-Eastern Poland, Northern Belarus and South-Western Lithuania. Old Prussia had also suffered numerous viking raids and, as in most Baltic countries, some viking settlement took place.

As for the German old Prussians, they were a mix of a large amount of settlers from all over Germany, as well as some settlers from the Netherlands. However, the local Prussians were not driven out nor were they all killed, although I am not sure as to the extent of Germanisation of the locals.

To a lesser extent, Prussia also took in some waves of Polish (predominantly Mazovian) and Lithuanian settlers, who would blend in with the rest of the society.