First off, some background on Spandau Prison. It was picked by the Four Powers (USSR, USA, Britain, France) as the place to hold the seven men sentenced to prison by the International Military Tribunal. Hess (who had been Hitler's deputy until 1941), together with Erich Raeder (head of the Navy until 1943) and Walther Funk (former minister of economics), had been sentenced to life in prison, while Speer (former minister of armaments) and Baldur von Schirach (headed the Hitler Youth before becoming Gauleiter, or district leader, of Vienna) were sentenced to twenty years, Constantin von Neurath (foreign minister until the late thirties, later was in charge of parts of the former Czechoslovakia until being replaced by Heydrich) - to fifteen, and Karl Doenitz (commanded the Navy from 1943-45) - to ten. The Four Powers had to unanimously agree to the littlest change made in the prison, and with the onset of the Cold War, the USSR ended up usually being the one to use its veto power.
Already in 1951, the USA began to release many of the war criminals it held, and the other Western powers followed suit, followed by the USSR releasing all of its POW's in 1955 as a conciliatory gesture from Khrushchev. When it came to Spandau, though, no release appeared to be in the cards, as these seven men were the faces of Nazism sentenced before the eyes of the world. The USSR especially blocked any attempt to free them and most attempts to make their lives easier, both because it had suffered the most during WW2 and so that it could claim moral superiority over the West, where several cases of high-profile Nazis getting high positions in the government or military fueled the flames of Soviet complaints about West German revanchism. However, they were not always the ones blocking releases. The British did all they could to have Doenitz's sentence be counted from the day of sentencing rather than the day of arrest, as they were afraid of him re-joining the armed forces, and in fact it was the Soviets who suggested releasing Neurath for health reasons all of a sudden after staunchly arguing for years that no early releases could happen.
Now, we come to the case of Hess, one of the three sentenced to life imprisonment. While Raeder and Funk were released in the fifties mostly for geopolitical reasons, as a gesture of goodwill from the USSR, they had legitimately been in poor health and it had appeared reasonable to many that they could die soon. Hess had no such issues. While his mental health was dubious on a good day, the experts who examined him deemed him not mentally ill in a way that would make imprisonment unjust, and his physical health was fine.
Some background on Hess - in May 1941, he flew to Scotland claiming that he wanted to achieve peace between Germany and Britain, as he didn't want the two "Nordic" countries to be fighting each other, especially since the German attack on the USSR was approaching. The USSR were, predictably, very unimpressed with this, and even though the British simply arrested Hess, they were still worried about potential connections between the British secret services and Germany. Hess was eventually convicted of conspiring to wage and waging aggressive war. Since he had been Hitler's deputy, his signature had been on all sorts of important documents, from the Nazi race laws to rearmament. While he hadn't been around to become personally responsible for the various atrocities in the USSR and the genocide of the Jews, he had set the scene for those that would. Additionally, it was his ideas about geopolitics and living space that resulted in the invasion of the USSR.
In 1966, Speer and von Schirach walked out of the prison, having served their twenty-year sentences to the day thanks to Soviet stubbornness, but Hess remained behind, the Soviets having blocked yet another attempt to release him. While everyone could see the absurdity of keeping an entire prison open for one inmate, the Four Powers couldn't agree on any other option, and so Spandau remained, the only other Four-Power organization in the world besides the Berlin Air Safety Centre. It played a very important role in world politics.
A Soviet warden who served there in 1985-87 remarked that nowhere else could a Soviet and an American sit down and chat about politics while playing chess and sipping coffee, but it was the governors' meetings were the real politicking occurred. As the Cuban Missile Crisis raged, the governors debated for hours whether Hess should be forced to get out of bed if he refused to do so of his own will, and whether they needed to buy a new elastic bandage for von Schirach.
At the end of the day, Hess remained an unrepentant Nazi. Even when he finally asked for clemency, he only cited his old age and deteriorating health and did not express remorse. He was openly racist to one of the wardens, which caused endless conflict. While more and more people pressed for his release, often fueled by the mistaken assumption that he was being held in solitary confinement (in reality, he talked to the wardens, could go outside for decent amounts of time, and could watch TV with the wardens starting in the seventies) and due to his "ascending age and descending health" (how Norman Goda described all of the inmates in his book), the USSR remained firm. Even the people who had worked to put Hess behind bars began to call for his release, irritating the USSR further. Until Hess' suicide in 1987 at age 93, they blocked every single attempt to free him.
Sources: Tales from Spandau by Norman Goda and Tajna smerti Rudolfa Gessa[The secret of the death of Rudolf Hess] by Andrej Plotnikov.
Flying_shadow gave a great and detailed answer, but I want to emphasize the Cold War dimension here. It's very important to note that Spandau was in West Berlin
As I'm sure you're aware, post-war Germany was divided into East and West. Berlin, within East Germany, was itself subdivided into East and West giving the Western powers (US, UK, France) an enclave within Soviet-occupied East Germany. Spandau, within West Berlin became a Soviet enclave within the Western enclave within the Soviet enclave. Sort of like Russian nesting dolls.
The arrangement at Spandau was to rotate among the four powers on a monthly basis -- each month, a new power ran the prison (so one month for the Soviets, one each for UK, US, France). As soon as Hess killed himself, the British razed the prison (it was technically in their sector of Berlin) to prevent the Soviets from insisting on continuing their presence.