In some of the 'conquered' countries, especially those in Central/East Europe, the German Nazis indeed had difficulty telling who was Jewish and who was not. In Poland, they had to rely on the help of the local Poles, who would obviously know which of their neighbors was Jewish. Many accounts talk about this. In From the Ashes of Sobibor, a Jewish survivor recounts how Germans had difficulty distinguishing the Jews from the general population, and so the Poles would voluntarily point out to Germans who the Jews were. He also recalled how Poles would catch Jews who were trying to escape or hide, and turn them into the Nazis. In the controversial book Neighbors: The Destruction of the Jewish Community in Jedwabne, Poland, Jan Gross also discusses the extensive and active way in which a large number of Poles cooperated with the Nazis. For many reasons -- fear of German reprisals, a long tradition of serious anti-Semitism, the desire to acquire the property of Jews -- many Poles participated eagerly in anti-Semitic initiatives, at times going so far as to participate in 'eradication'.

In Germany itself, the Nazis could rely on records. Germany was an organized and bureaucratic state which kept extensive marriage and birth certificates, which could be used to trace Jewish ancestry. It was these files which allowed the Nazis to be so precise in their racial categorizations; having even one grandparent who was Jewish made you non-Aryan. In Albert Speer: His Battle With Truth, it was noted that Hitler had to fire a secretary after she was found to have a Jewish grandparent. Hitler was reportedly wistful, since the secretary had been a good secretary, but he claimed that it was important for him to set a good example.

Note that I am not a historian; I am simply someone who is very interested in this topic. I hope this response has been helpful.

Follow up question. Was there many Jews who pretended not to be Jewish to escape persecution?

To give an answer specific for the Netherlands;

The country back then was in a state of pillarization, meaning politico-denominational segregation; in this case essentially based on religion. This meant that each pillar had their own schools, banks, trade union, political parties, sport clubs and so on.

When the Germans invaded and took over they issued and created, together with Dutch collaboration, the identity card which everyone had to carry with him or her. For Jews, this meant an added 'J' on their card.

This, combined with the pillarization meant that different religious and ethnic groups were essentially seperate. Therefore, both Germans and collaborators were able to produce stippenkaarten (lit. dotted maps) such as this one for the city of Amsterdam with the number in blue meaning the Jews per neigbourhood and every dot meaning '10 Jews'.

Follow up, although it might be weird: were there any instances of Jewish people hiding their ancestry and actually joining the Nazis?

Not to discourage further posts, but this older r/AskHistorians post by u/OldWorldGlory links to several good answers to both the OP's question as well as some of the follow up questions posted in this thread: https://www.reddit.com/r/AskHistorians/comments/2wj741/how_did_nazis_identify_who_was_jewish_or_not/

Follow up: Many movies seem to depict a check for circumcision to indicate a hidden Jew. Did this actually occur?

Follow up question: how would this identification have progressed, had Germany succeeded in the war?

Recenly had a powerful experience via finding out that, genetically, I come up as Ashkenazi Jewish. Following that, I read in the sci-fi novel The Man in the High Castle, that blood tests are performed to identify Jews in America as part of the story. This seems like a believable next step.

Where were we at in terms of identifying ancestry via genetic testing by the end of the war? It's hard to say "what would have happened," so I'll ask instead, "How close were we to this technology?"