But it's curious how Yiddish became the lingua Franca for so many Jews who presumably never settled near German speaking Europe.

Here's the issue--they did! Ashkenazi Jews, who are the Yiddish-speakers, migrated from Central Europe to Eastern Europe centuries ago. The exact origin of Yiddish is debated (the main debate is whether it originated in the Rhineland or along the Danube), but the consensus is that it's in somewhere German-speaking (there is a hypothesis that Yiddish isn't actually Germanic...but it's rather fringe). Due to social segregation the Jews kept speaking it, though Yiddish was heavily influenced by Eastern European languages. A bit of colloquial early 20th century Yiddish would probably be tougher than what I assume is the rather standard literary Yiddish you're listening to.

Outside these groups, Yiddish isn't a lingua franca. Jews from areas that aren't central or eastern Europe generally don't speak Yiddish, and historically it doesn't have much use outside people from that area.

So the point is that Ashkenazi Jews are from areas larger than modern Germany, because of historic migrations.