Yes, I know Hess was there longer
The immediate answer was that of the "Spandau Seven," von Schirach was comparatively younger (bn. 1907) compared to the three prisoners released earlier before their sentences were up- von Neurath (bn. 1873), Raeder (bn. 1876), and Funk (bn. 1890). All three of these men developed notable health issues within their incarceration. These health issues dovetailed with an active clemency body within the FRG for the early release of IMT convicts. Dönitz only received a ten-year sentence and was released on time. Von Shirach did develop a blood clot in his arteries that might have allowed him a slightly earlier release, a fact that rankled Albert Speer who also had the same sentence. But the Spandau authorities released both men on the same day, 30 September 1966, when their sentence was up.
The larger reason for why von Schirach and Speer had to serve their sentence and why Hess had to linger on until the 1980s was the breakdown of inter-Allied unity in the immediate aftermath of the war. The Allies made wartime agreements that Spandau and the IMT convicts would be the collective responsibility of Allies. Four representatives of each country (US, UK, France, USSR) governed Spandau and a complex set of prison management evolved around this clumsy relationship. The structure of the wartime agreements over Nazi war criminals convicted by the IMT made it so that collective agreement was needed for either early release or other changes with the management of Spandau. This was in stark contrast to the relationship that evolved with war criminals convicted by the Subsequent Nuremberg proceedings and various national trials of German offenders. Here the FRG government could conduct bilateral talks with the jailing authority to gain clemency. Bonn found it got much traction out of the US, for example, in light of the Cold War and most of the convictions of the American NMTs were either overturned or prisoners amnestied.
Such "amnesty fever" was not going to happen in Spandau with the Soviets having a veto over the process. The Soviets and the GDR learned very early on in the Cold War that Western softness on Nazi criminals was a valuable propaganda point and they were not going to cede this advantage. Having the Soviets in Spandau hold the line against the amnesty fever in the West was valuable moral high ground the Soviets would not surrender. In short, it cost the Soviets nothing to maintain a hard line on Spandau while they could simultaneously portray the West as soft on Nazism. Potentially reversing themselves on Spandau was also a chip the Soviets could use with Bonn in its semi-clandestine dealings with the FRG by holding out the potential of amnesty in exchange for some other concession. Soviet officials were appreciative that health problems could make these men martyrs, hence the early releases. But this left the other Spandau inmates who were in relatively good health with little choice but to serve out their sentences.