First of all, in Germany itself Jews overwhelmingly spoke German, not Yiddish. Most of the Jews killed by the Nazis did speak Yiddish, in Eastern Europe, but in Germany itself Jews had largely switched from Western Yiddish to German.

At various points in modern European language was used as a marker of ethnicity. In Austria-Hungary, the government had an interest in inflating the number of Germans, so they counted Yiddish as German in censuses, and used that to emphasize the number of Germans.

But, Hitler (and the Nazis) tended to use criteria about ancestry for defining who was Jewish (and who was German). That's largely because they did not base their antisemitism predated coming up with logical-sounding reasons for it.

Besides, it wouldn't've made sense to define Yiddish-speaking Jews in Eastern Europe as Germans. Their ancestors hadn't lived in Germany for centuries, and when they did they weren't "native", so to speak. Either way, including Jews as Germans wouldn't've made sense because then the Nazis wouldn't be antisemitic, and that's an important part of what made the Nazis the Nazis.