I saw your question a couple of days ago and I have been doing some research since then because initially, I could not find anything for your title question.

The bad news is, I still couldn't anything coming from Jewish Holocaust survivors on the matter of this debate. As for Jewish survivors debating German practices of dealing with the National Socialist past, I think the texts by Jean Amery and Jospeh Wulf would be the most pertinent. Especially Wulf, who was a pioneer in terms of German historical research on the Holocaust and who is responsible for opening the memorial in the Wannsee Conference House makes a very good reading on the broader strokes of West-German memorial culture (he once famously wrote: "With the German you can document yourself to death - you can have a democratic government in Bonn. The mass murderers are still free, have their houses and grown their flowers")

Also, I found that when Speer was released, Erich Goldhagen, professor, holocaust survivor, and son of Daniel Goldhagen, attacked Speer's version of his own guilt by saying he had been there when Himmler had given his Posen speech and thus knew about the Holocaust. Speer denied everything but this never made it too far in the German public.

As for the debate itself, the important matter is that the Germans who pressed for the release of the last seven Nazi criminals held in Spandau were never going to be successful. Spandau was run by the four war allies together and despite a certain tacit readiness for the release of the prisoners on the side of the Western Allies, the Soviet Union pretty much blocked any early release and the Western Allies would not push the matter since they were afraid it would lead to another Berlin crisis. (See: Norman J. W. Goda: Kalter Krieg um Speer und Heß: Die Geschichte der Gefangenen von Spandau)

On the German side, the arguments were varied. The German government stressed foremost that the prison was a. anachronistic since they felt Germany as a nation had rehabilitated itself in the world and keeping the last three (by this point in 1966, it was only Schriach, Hess, and Speer left in there; Funk, Neurath, Raeder and Dönitz had been released more or less without problems) was just not in accordance with this image of Germany they wanted to project. And b. the other argument they brought forth was that it was expensive. The German government resp. the West Berlin government had to pay a share of keeping Spandau prison operating and they basically bemoaned that keeping up a prison for 1000 people to just imprison three was expensive and impractical.

When it came to moral arguments, one that was brought forth regularly, for example by Der Spiegel for releasing Hess was that keeping Hess in Spandau was assigning him "greater importance than he ever had as a Nazi". Another argument from the Konservative Aktion, an organization of right-wing CDU and CSU members, campaigned for Hess' release with the argument that he was an old man and keeping him in there served no purpose since he and Germany had paid for their crimes.

The only supporter of the release of Hess, Speer, and Schirach I found who was also a victim of the Nazis was Martin Niemöller, protestant pastor and author of the "first they came for the..." text. Niemöller in 1963 started a public campaign for their release. In this Der Spiegel Interview he also used the issue of financial costs as well as the fact that they were sick and old and he felt it was needed to release them in order to help their families cope with them.

In essence, the whole thing never got very far and I believe that the reason why Holocaust survivors never joined the debate was that any debate had in Germany was doomed from the start since the USSR refused to release the imprisoned of Spandau before their term was up and the Western Allies were not interested in pressuring the Soviets for political reasons.