There were seven people sentenced to prison by the International Military Tribunal, four to terms of years and three - to life. They were under Four-Power control, meaning that the USSR, USA, UK, and France worked together in keeping Spandau Prison running. They were the only prisoners held in such an institution. The others had been convicted under the jurisdiction of specific countries. The United Stated began to let theirs go already in 1951 (the aptly-named 'amnesty fever'), thanks to the Cold War making them want to have West Germany on their side. In 1958, the last two were released. The USSR didn't pursue a similar line with East Germany, so they only let the last of theirs go in 1955, when Khrushchev used them to improve relations with the West. These last 10,000 returnees were everyone from serious war criminals sentenced to life imprisonment to unlucky POWs who had never been convicted of anything. So to put it quite simply, release had nothing to do with justice and everything - with how the political winds were blowing.

The seven men of Spandau, however, were out of luck there. There was a big difference between the major war criminals convicted before the entire world and an anonymous low-ranking officer who burned a village to the ground. I'll go through them in order of release. Konstantin von Neurath had been sentenced to fifteen years, but he was released in 1954 because of his legitimately bad health. The Soviets decided that the best thing to do was to let him go home and die quietly, which he did not even two years later. In 1955 came the turn of Erich Raeder, who had been originally sentenced to life and was released for similar reasons. The following year, the ten-year term of Karl Doenitz was up. The British and Soviets had both kept him in there as long as possible, counting the term from day of sentencing rather than day of arrest, because they worried that the former admiral would get his old job back. However, he was released without issue. In 1957, another lifer, Walther Funk, was released because of his bad health. After that, Albert Speer and Baldur von Schirach were forced to wait until 1966 for their twenty-year terms to end. The Soviets refused to even hear of letting them go.

So what of Hess? Hess simply had the bad luck to be sentenced to life and in good health. Rumours that he was insane were unfounded - he wasn't exactly mentally well, but he was perfectly capable of serving his punishment. And the fact that he was Spandau's sole inmate did not help matters. The Soviets needed Spandau as a foothold in West Berlin, so they weren't interested in having it close down. A part of their propaganda was that all the Nazis were back in power in West Germany, so holding on to Hess to the last gave them the chance to say 'Hey, look, we're trying to hold this Nazi in prison but the West wants him released!' In 1987, at the age of 93, Hess killed himself and by doing so launched a thousand conspiracy theories - but that is an entirely different story.

He, however, was not the last Nazi in prison - check out this answer by /u/sickhobbit about the Breda Four, Nazis held in a prison in the Netherlands two of whom were only released in 1989.

Source: Tales from Spandau by Norman Goda.