Ethnic identities in Eastern Europe before the Second World War are difficult to parse for a couple of reasons.

To begin, you should remove this notion of a more-or-less ethnically homogeneous nation-state from your mind (that is, if it was there in the first place). It simply did not exist in Eastern Europe before the Second World War. Rather, what you had were territories in which so-called language islands existed, in which small areas would be populated with speakers of a specific language, right next to other areas of speakers of a different language. And the mixtures would be all over the place - so you'd have German language "islands" in Poland, or Czechoslovakia, or Hungary, or Romania, etc. You would have Romanians in Bulgaria, Russians speakers in Poland, and Polish speakers in Germany. As you mentioned, there were, of course, Germanization campaigns in Eastern Prussia, but they could only be so effective.

But language islands were not the only feature. Many people in Eastern Europe were bi- or even tri-lingual. German and Russian tended to be the languages of government in Eastern Europe before the First World War, so many people, regardless of their native tongue might learn one of those two languages. The same goes for native German speakers if they lived primarily among speakers of another language, like Polish. Jews, who in some parts of Eastern Europe made up significant percentages of the population as a whole, would often speak Yiddish, along with German and perhaps Russian or Polish as well, or one of the other languages that were predominant in their local area.

But we must also not forget that one's native tongue is not always an adequate identifier of ethnic identity in Eastern Europe. Some people didn't identity with a specific ethnic group at all, rather in answer to questions on their ethnicity would state that they are "like the people from here", and that they "speak the language of the people here" (this is meant to be in contrast to someone who might answer that they speak Polish and are from Poland). This is a sort of willful lack of identifying with a particular ethnicity. Further, even if the person had a particular ethnic identity, it might appear to be very confusing to our eyes. A perfect example is that of the "Three Smiths" from the Lithuanian town of Marijampolė, just outside of the then German Reich (but very indicative of the region generally). There were three families, one named Schmidt, one named Kowalski, and one named Kusnjetzow (all three of which mean "Smith" in German, Polish, and Russian respectively). Mr Schmidt, with the German given name of Heinrich, identified as Polish (an ardent nationalist, at that), while Mr. Kowalski thought of himself as Russian (despite his Polish name) and Mr. Kusnjetzow identified as German. Their confessional identities didn't clear things up either, as Schmidt was a Roman Catholic, Kowalski was a Russian Orthodox, and Kusnjetzow was Evangelical.

So how German were the areas of Eastern Europe controlled by the German Reich? The best overarching answer I could give would be, it depends. There were definitely some areas that would have been thoroughly German, but those areas would have been mixed and interspersed with areas that were not at all, and all of these areas would have had people who had ethnically ambiguous identities.

A couple of sources you might be interested in are: Tara Zara, "Imagined Non-Communities", and Vejas Liulevicius, War Land on the Eastern Front.

EDIT: The Liulevicius book has been translated into German as Kriegsland im Osten, but I think the Zara article is English only.