Your question needs a bit of clarification--do you mean distinct as opposed to their non-Jewish neighbors, or distinct as opposed to Jews in other countries? Jews in Poland did have a distinct language from their non-Jewish neighbors, they spoke Yiddish. Or are you asking why they spoke Yiddish, and not Judeo-Polish or something?

If you're asking why their language wasn't different than Jews in other countries, most of it is naming conventions. Different Judeo-Romance languages get to be languages they didn't all have a unified literary tradition, they get separate names. In reality, both Judeo-Spanish and Judeo-Portugese share some features as opposed to Spanish and Portugese, respectively, which reflect their complex dialect continuum that existed in the Middle Ages. Communities sometimes used a mix--the Bevis Marks synagogue in London switched between keeping its records in Portugese and Spanish, for example. In an alternate history, they'd simply be considered dialects of Judeo-Iberian or something. And there are several different varieties of Judeo-Spanish, which diverged in different places Sefardic Jews moved to--North Africa vs Turkey vs the Balkans (vs Western Europe, but Judeo-Spanish in Western Europe is pretty thoroughly extinct I believe)--which each could just as easily be called separate, but closely related, languages.

And on the other side of things, Yiddish dialects are actually significantly different. While formal written Yiddish generally has long been in some sort of vague standard (with people pronouncing that standard differently), vernacular Yiddish tends to borrow very freely from local languages. I've heard (anecdote warning!) that speakers of Romanian Yiddish and Lithuanian Yiddish have trouble communicating, because each has an extensive set of loanwords that the other lacks. But they could each read each other's newpapers. Now the dialect situation is different entirely--the destruction of Eastern European Jewish communities by the Nazis has upended the few remaining pockets of Yiddish speakers, so the loanwords from Slavic languages have largely been replaced with Modern Hebrew or English.

But, Polish Yiddish didn't match the borders of Poland, because borders of Eastern European polities shifted a lot over history, and the borders of Jewish cultural regions didn't move to match. But, there is a Polish Yiddish, which is distinct from other dialects of Eastern Yiddish (to say nothing of Western Yiddish, spoken in Germany and the immediate area, which was even more different), and historically it borrowed lots of Polish words.

If you're asking why they kept speaking a Germanic language, rather than developing Judeo-Polish, the answer is probably prestige. German was a prestigious language, and the German Jewish community was a prestigious one in the Jewish world (at least, in their own minds). In Eastern Europe, where many different languages coexisted, it isn't so surprising that Jews wouldn't adapt to the local language. This would've worked partly because Jews would've seen using Yiddish (like the rest of the Ashkenazi world) as more prestigious than using Polish, which speaks to a degree of social segregation. But the volume of Slavic loanwords and other features in Eastern Yiddish speaks to the fact that speakers of Yiddish and Polish (or Russian, or Lithuanian, or Ukrainian, or Hungarian, or Romanian) were not isolated. Certainly a large number of Jews spoke the local language fluently, as a second language (though not all).