Prior to World War Two, there were millions of ethnic Germans living in the east, both inside Germany's borders and outside. Not only were there many Germans in Poland, but Czechoslovakia also had a huge German population dating back to the Medieval Period, and others lived even further east in Latvia, Estonia, Russia, and Romania, among others.

After the war, many of these Germans were expelled or fled west, particularly those in Czechoslovakia and Poland. Of course even if things like the Beneš decrees which expelled Germans in Czechoslovakia tried to make a simple split between who was and was not a German, it wasn't really that simple in regions like the Sudetenland where Czechs and Germans had been living side by side for centuries. Jews who had integrated into urban German society were another case which complicated the process of expelling all Germans. There are certainly some cases of individuals staying, but most were expelled.

Poland is a bit of a special case because as a result of the war, they lost substantial territories in the east which had Ukrainian and Belarusian majorities but significant Polish minorities, all while gaining German land in the west. Many Poles from these eastern regions ended up moving into the new western ones, for example in cities like Gdansk/Danzig.

There are still people in Germany from these expelled communities advocating for more recognition of what is essentially a massive case of ethnic cleansing.

I have more sources on the history of Czechs and Germans than Poland, but they are a good resource for this topic. I would recommend these two books for more information:

Budweisers into Czechs and Germans: A Local History of Bohemian Politics, 1848-1948 by Jeremy King

Kidnapped Souls: National Indifference and the Battle for Children in the Bohemian Lands, 1900-1948 by Tara Zahra