This is the subject of a great deal of debate. A big part of the issue is that we can't simply ask old Germans if they knew what was going on to Jews. Well, that's not quite true--we can ask them, but we can't be sure they're telling the truth. While we tend to think of Germans as a group as feeling guilty about WW2, this does not necessarily mean each individual German who survived that era feels that way about their own actions.

It is important to note that the mass killing of Jews happened almost entirely in Eastern Europe, not in Germany proper. But, there would've been strong indications something was amiss. Jews were being rounded up en masse and sent for "resettlement in the East", a common euphemism. If you put two and two together, the fact that many thousands of Jews were rounded up and shipped "to the East" and never heard from again, you could reasonably deduce that something bad was happening to them. But you might not have jumped straight to mass murder, and humans are really good at not putting two and two together if the result of that is distressing to them. And the Jewish population in Germany was relatively small, especially after a large share fled during the 1930s. So to most people, they wouldn't've seen the full extent of the roundups in the first place.

But, people do talk to each other. There were rumors about what was going on in the East--people who had, say, entire families and Poland or Ukraine and then one day stopped hearing from them and was unable to contact them or anyone who knew them was aware that something was wrong, and they told people. Word about mass murder spread, but at that point it was just rumor, and as above people will often choose to not believe something so distressing if possible (which is part of why the allies often forced Germans to bury bodies in camps near them, forcing them to come face-to-face with the atrocities).

And the extent of the massive genocide machine that Germany produced meant that there were a ton of Germans who would've been at least peripherally aware of what was going on. First of all, there were some camps in Germany. Not Death Camps, where the sole enterprise of the camp was mass murder, but camps were brutality and murder were the norm. Certainly neighbors of Mathausen, Buchenwald, or Bergen-Belsen would've known people were being killed in large numbers there, or they should've known and intentionally remained ignorant. I mean, over the years one of the guards will come into town and let something slip, someone will take a wrong turn and see barbed wire with starving people behind it, etc, to say nothing of people who actually lived near these places.

But, part of the massive genocide machine meant that a huge number of people were involved in other ways, not just being a guard in or living near a camp. Anyone who worked for the railway would have seen trains full of Jews crammed into cattle cars and not returning. While a German knowing their neighbors were "deported" might reasonably be ignorant of the scale and brutality of what was going on, seeing thousands upon thousands of people shipped east couldn't be. And for, say, the train engineer actually arriving at a camp, he would've seen thousands and thousands of people constantly arriving at a camp that clearly lacked the size or facilities for even one trainload of people, let alone daily trains of people arriving. Germans were involved in processing and distributing Jewish possessions (jewelry, hair, shoes, clothes, etc), which also should've raised eyebrows. Engineers had to design vans for gassing people, and from records of the R&D process it is clear they would've known that these were used in practice, not just in theory, to kill groups of unsuspecting people.

And that's before we even touch the subject of regular soldiers on the Eastern Front. The Wehrmacht committed a litany of war crimes in Eastern Europe, which included indiscriminate murder of civilians. While not every soldier actually committed these war crimes, most would've been aware of them. And they would've been in close proximity to active genocide, seeing huge populations crammed into ghettos that clearly couldn't fit the population, people "deported" from those ghettos, etc.

Many of those who were aware and not deliberately ignorant would've talked about this to their friends and neighbors, even if they didn't mention specifics. It is therefore very likely that a significant segment of the German population was aware that something horrible was happening, even if they didn't know details. It is impossible to accurately guess what share of the German population that was.