Most of them got away with it, quite frankly. According to Günter Lewy, fewer than 10% of the people responsible for Nazi crimes were actually prosecuted after the war, and that of those prosecutions, only about 7% were directly related to the Holocaust; Lewy calculated that there were fewer than 1,000 prosecutions for Holocaust crimes in West Germany (out of some 86,000 people he identified as direct perpetrators, i.e. members of the SS-TV, Einsatzgruppen, and various police formations). Many of the rank-and-file members of the SS and police lived unmolested in postwar Germany for decades without being investigated or prosecuted, and some of them even continued to work as police officers in the post-Nazi police forces. Some perpetrators were actually arrested by the Allied occupation forces and held as prisoners of war, but were not identified as war criminals and were released, only for their identities and responsibility for mass murder to be uncovered later. For example, Erich Bauer, a low-ranking member of the SS who operated the gas chambers at Sobibór, was captured by the Allies after the war, but was released in 1946 and returned to live in Berlin; he was only arrested in 1949, after a Sobibór survivor identified him after a chance meeting at a fair.

So, obviously, some of these people were later discovered and prosecuted. The West German courts continued investigating and charging Nazi criminals into the 1970s, but as the records of the responsible investigative branches housed by the German Federal Archives in Ludwigsburg illustrate, by the time many of the perpetrators were identified, they had already died, and in many other cases the German investigators weren't able to find them. In the cases where people were prosecuted, they were generally charged as accessories to murder rather than for murder, which yielded lighter sentences. This was the case even at some of the more prominent Holocaust Trials (such as the Frankfurt Auschwitz Trials from 1963-1968), where many of the defendants were given sentences of only a few years; the Auschwitz Museum notes that fewer than 800 of the more than 8,000 known Auschwitz personnel were ever tried. Lewy attributes these relatively short sentences to a faulty understanding of Holocaust responsibility, where Hitler, Himmler, Heydrich, et al. were the truly responsible parties, and those below them were "just following orders", and were only accessories to the crimes of the higher-ups.

There are also the famous cases of people who managed to escape Europe altogether, only to be detected decades later. Of course there's the prominent example of John Demjanjuk in the United States, who was accused of being a guard at Treblinka. It was never established officially whether he was "Ivan the Terrible" of Treblinka (that's a whole other rabbit hole I don't want to go down here), but he was almost certainly a guard at Sobibór, which he was convicted of in Germany; however he died while his case was on appeal, so he was officially still innocent of the crimes he was accused of. There were a number of other cases of perpetrators who moved to the US and Canada after the war and later identified as war criminals; some of them were deported and prosecuted, while other cases spent decades winding their way through the courts without resolution. This is a bit off track from your actual question, but it's an interesting subject in its own right.

I can also speak to the case of Romania, since that's actually what I'm researching at the moment. In Romania, it's even more true than in Germany that the vast majority of people got away with it. It's a different situation because of the fact that Romania was under Soviet occupation and the trials took place in the context of a political transition to a Communist state, but very few low-level perpetrators were ever brought to trial; some of them were captured by the Soviets and tried in the Soviet Union, but the majority of the trials that took place in Romania were of high-ranking government and military officials, along with journalists and other political figures, rather than the actual triggermen who did most of the killing. Even the top-level Romanian perpetrators got off relatively easily; only four people were executed after their convictions in the postwar trials, and even the high-ranking officers who were convicted were mostly amnestied in the 1950s and 1960s.

The trials of low-level Holocaust perpetrators are really an under-researched area. The records are out there (I've worked with them myself), but they're largely untapped. A lot of the focus has understandably gone to the Nuremberg Trials and other trials of prominent political and military leaders and high-level Holocaust perpetrators without really examining what happened to the people who actually did most of the killing. Hopefully that's going to change in the near future as more and more records become publicly accessible.

My main source for this answer was Günter Lewy, Perpetrators: The World of the Holocaust Killers (Oxford UP, 2017), which is a good place to start if you're interested in the low-level perpetrators of the Holocaust. I also drew on BA-L B 162, the postwar trials collection at the Bundesarchiv in Ludwigsburg, Germany (much of which is digitized and available for research in the US at the USHMM), as well as the Final Report of the International Commission on the Holocaust in Romania for my comments on Romania.