Your question is really two questions: "Why did communist governments in Eastern/Central Europe cease to exist?" and "Why did the USSR collapse?".
There are three parts and multiple posts for this answer, so bear with me!
Part 1: Poland
1/2
[the following posts are an abbreviated version of lectures I give. The text has been assembled over a period of years, and contains information and commentary from a wide range of sources. I've tried to give a bibliography that represents the scholars I have used]
On question one, Poland is crucial as the nation that (other than the USSR) that does the most to instigate the collapse of communism. It had always been the most rebellious Warsaw Pact member. In July/August 1980, a wave of strikes broke out after massive food price increases. To try and deal with this, the Polish government officially accepted the communist world's first independent trades union: Solidarity.
In exchange for recognition, Solidarity’s leader, Lech Wałesa, acknowledged the communist Polish United Workers’ Party (PZPR) primacy, contending that Solidarity did not seek to become a political organization and that their aim was engage in meaningful reform.However, the social and economic situation in Poland was such that it was unlikely a real partnership would emerge between Solidarity and the PZPR. Solidarity’s membership exploded to 10 million people in a population of around 35 million. Further strikes broke out, temporarily settled with wage increases. By December 1980, wage increases had reached 13% while the food supply had decreased by 2%. The gap widened into 1981. In each confrontation, the regime backed down. Deliberately or not, Solidarity became a major political organization. While it never claimed state power, it portrayed itself as the Polish nation’s representative.
Soviet leaders were hostile to Solidarity from the outset. Starting in late August 1980, they conducted military exercises on Poland’s borders, but Soviet leaders were reluctant to resort to direct military action. The Politburo had by June 1981 made a secret decision not to intervene militarily in Poland under “any circumstances.” The Soviets expected serious armed resistance and feared that significant components of the Polish army might end up fighting Soviet and Warsaw Pact troops. They also feared toughened international sanctions, in addition to those that had been imposed on the USSR as a result of the invasion of Afghanistan. This doesn’t mean that Moscow was willing to abandon Warsaw. They were ready to defend their bases/Warsaw Pact lines of communication, by force if necessary, but none of these plans for action, or inaction, was ever enacted.
Officials in Warsaw, including Prime Minister General Wojciech Jaruzelski, reneged on promises made to Moscow to introduce martial law. It was only after Jaruzelski became party leader, on September 6, that proper plans to impose martial law were made. Solidarity’s leaders did not take this threat seriously, convinced that the Communists wouldn’t dare enforce martial law. The union’s leaders were taken aback by Jaruzelski’s actions on December 13, 1981, where he imposed 'order' and arrested key Solidarity figures.
Solidarity's repression was largely welcomed in Moscow and in the capitals of other communist countries. This repression was seen as a reminder of Soviet thresholds of tolerance. However, a common thread in the revolutionary changes of 1989 was the gradual discovery of Soviet tolerance. Gorbachev's democratization measures in the USSR in 1987–88 were intended to legitimize the communist party’s leading role. Together with economic reforms, they were meant to lead to a new model of democratic socialism, not a form of capitalism.
In Poland, Jaruzelski had decreed an amnesty for political prisoners in September 1986, believing he had the upper hand over Solidarity, which continued to operate underground. After its re-legalization in the spring of 1989, Solidarity’s membership reached two million, only a fifth of the 1981 level. So Jaruzelski entered into negotiations with Solidarity with a reasonable degree of confidence. These discussions were intended to address the economy and to limit negative public reaction to the consequences of upcoming economic reforms.
When negotiations began on February 6, 1989, all issues were on the table (apart from foreign policy). The real issue was the PZPR's power. On April 7, 1989, agreements recognised Solidarity not only as a trade union, but also as a legitimate political opposition force. A crucial point for Jaruzelski and his subordinates was the Solidarity leadership's agreement to participate in elections. These would be conducted according to rules that in large part served only to preserve the PZPR’s authority. Opposition parties would compete for 35% of the 460 seats while the other 65% would be left unopposed to the PZPR and its subordinate parties. The new 100-seat Senate (a legislative body with far less power) was to be elected at the same time. However, to override a Senate veto, a two-thirds’ majority would be required in the lower house. Consequently, the party might have to negotiate with the opposition on some of the government’s programmes. The two houses of parliament sitting in joint session would elect the president, who was to wield considerable powers. Given the far greater numbers in the lower house and the PZPR’s officially sanctioned dominance, the formula ensured Jaruzelski’s continued power.
The official Soviet reaction was enthusiastic. While Gorbachev was not prepared to accept multi-party politics in USSR itself, the Polish agreements fitted well with the Gorbachev’s domestic and foreign-policy goals. While the communists retained power, the democratization process was credible enough that Western countries might look kindly on it. This was - in Gorbachev's eyes - a significant step toward reconciliation between the two Europes, and necessary step towards further arms reductions and wider East-West cooperation. However, just two months after the Polish agreements, things started to fall apart.
1/2
Part 2: East Germany
[the following posts are an abbreviated version of lectures I give. The text has been assembled over a period of years, and contains information and commentary from a wide range of sources. I've tried to give a bibliography that represents the scholars I have used]
The changes in Poland in 1989 that I covered in the previous post has been described as a “negotiated revolution.” This revolution was initiated from above by political elites. Subsequently, revolutions "from below" occurred in East Germany, Czechoslovakia, and Romania. The crucial moment between the two forms of change was the Berlin Wall’s fall. So, how and why does the Berlin wall – the symbol of the Cold War – collapse?
Throughout the 70s and 80s, state repression in East Germany through the Stasi secret police increased, and dissatisfaction inexorably grew. East Germans declining hope for liberalisation coupled with the disintegrating image of a looming "capitalist threat" created widespread dissatisfaction. Fear of 'the West' collapsed in the face of smuggled video tapes, West German TV broadcasts, pirated pop music cassettes, and glossy magazines. While other nations in the communist sphere were liberalising, the GDR’s Sozialistische Einheitspartei Deutschlands (SED) under long term leader Erich Honecker only increased its grip. Jails were filled with those who had tried to leave the country, and the police beat anti-regime protesters on the streets of East Berlin. Despite the repression, many opposition groups, citizens’ committees, and new political parties formed, meeting in private or in the shelter provides by churches. Unable to flee west, the GDR's citizens began seeking refuge in the West German embassies in Budapest, Prague, and Warsaw.
On September 10, 1989, Hungary opened its border with Austria to the refugees, while thousands of GDR citizens were crowded into the West German embassy in Prague. West German Foreign Minister Hans-Dietrich Genscher addressed the crowd and announced that sealed trains would take them through the GDR to the west. As the trains passed through Dresden, the GDR's Volkspolizei beat the thousands people trying to climb onto the trains. Amid protests in Leipzig, Berlin, and other cities and towns, productivity dropped like a stone (and a very heavy stone at that) and the GDR approached economic collapse.
Dependent on Soviet energy supplies and Western credit, the GDR's command economy could not meet its most basic obligations. Shortages of vital, everyday commodities grew and grew, and the brutal realities of everyday life contrasted with the Party's propaganda. The Party and the state's legitimacy crumbled as the situation worsened. Popular protest and the dire economic situation provoked disquiet and moves to unseat a leader when a mixture of loyalty and distrust kept a lid any revolt within the Party.
Approaches to Gorbachev invariably brought the response that any personnel changes had to be achieved by the SED themselves. Finally, a small group led by Honecker’s protégé, Egon Krenz, called for the dismissal of Honecker. On October 18, 1989, Honecker agreed to be replaced by Krenz “for health reasons.” The new leader promised a new era in dealing with the concerns of the people. He announced steps to legalise new political parties and draft a more liberal travel law.
With great fanfare the SED announced further reform. They might have succeeded had the reforms come earlier, but by now all confidence in the leadership had evaporated. SED spokesman Günther Schabowski’s garbled announcement of changes to travel regulations that had been approved by the Politbüro on November 9, 1989, finally forced the Wall open. When asked by a journalist when the new rules would go into effect, Schabowski, with only fragmentary information to hand, said “sofort [right away]”. He did not mention that passports and visas were required. When people heard they could visit the West without any formalities, they rushed to the transit points in Berlin, overwhelming the confused and frightened border guards.
Thousands of deliriously happy East Germans filled the streets of West Berlin, welcomed by their fellow citizens with cheering and drinks. These dramatic events took place right on the doorstep of the Soviet Embassy located near the Brandenburg Gate. When news of the opening of the Wall reached Moscow, Gorbachev was highly agitated. But Krenz reassured the Soviets that nothing dramatic had happened. He said the new travel regulations had gone into effect prematurely, but the government would soon be able to control events Of course, this was an outright lie; party and state structures were disintegrating.
Symbolic were citizen-led raids on the Stasi headquarters in Berlin and Dresden, where the streets were littered with once-secret documents. The greatest danger, though, was that, in a desperate effort to restore order, the East German military would intervene. In expectation of more demonstrations in Berlin, units had been mobilised, and it was known that in case of an emergency the army had plans to occupy West Berlin. Nothing of this sort happened. Although on November 10, a state of alert had been increased, the political and military leaders were no longer able to give orders that were heeded.
When the Soviet Union tacitly acquiesced to the fall of the Berlin Wall, it was a momentous event, providing dramatic and incontrovertible confirmation of the Brezhnev doctrine’s demise. In the two weeks that followed, as the East German regime began to crumble with not the slightest Soviet show of force, the magnitude of Gorbachev’s incremental revolution became clear.
Next: The Gorbachev Revolution and the USSR's Disintegration
Part 3: The USSR's collapse.
[the following posts are an abbreviated version of lectures I give. The text has been assembled over a period of years, and contains information and commentary from a wide range of sources. I've tried to give a bibliography that represents the scholars I have used]
As détente collapsed in the mid to late 1970s, so too did the dream of the USSR as a socialist paradise. Although it was still a superpower in terms of its nuclear arsenal and gigantic ground and air forces, it lagged behind the USA in pretty much every other measure. The USSR was also hamstrung by internal factors. It was led by a group of ageing, sick old men, wedded to increasingly outdated dogma. Leonid Brezhnev died in November 1982, to be succeed by former KGB chief Iiuri Andropov who died in February 1984, who was then succeeded by Konstantin Chernenko who died in March 1985. At the top, there was no innovation, new thinking, or desire for change.
The Soviet economy was in a terrible state, as the gargantuan military-industrial complex absorbed vast amounts of capital. Consumer goods were sparse, queues for food common, and it had become clear that the Kremlin's tales of how things were so much worse in the west were nothing more than falsehoods. Soviet satellite states in Eastern Europe - like Poland - were becoming restive, with demands for greater political and social freedoms. Further afield, Afghanistan was a running sore, consuming personnel and resources in the seemingly unwinnable war against the mujahedeen.
However, in the mid-1980s, dramatic changes occurred. The most important in terms of national leadership was the ascent of Mikhail Gorbachev to the top position in the Soviet Union. It became clear to the Soviet leadership that they could not continue with the 'revolving door' of ill old men continually reaching the top and then dying within a few months. On March 11, 1985, one day after Konstantin Chernenko's death, Gorbachev became general secretary of the CPSU and thereby leader of the USSR.
Prior to his ascent, Gorbachev had been identified by Western leaders – particularly the UK's Margaret Thatcher – as a high Soviet official that they could work with. Gorbachev was willing to debate and discuss in good faith. He was also accompanied everywhere by his wife Raisa, who charmed Western hosts with her intelligence, wit, and astute political sense. As a couple, the Gorbachevs were a radical departure from what had come before.
Gorbachev came to power as a reformer, not a revolutionary, and his reformist policies can be summed up in two words: Glasnost and Perestroika. Glasnost was his policy of openness, increased transparency with Soviet government, and a willingness to tolerate dissent and discussion. Perestroika was reform of the USSR’s politics and economy, but within a socialist system. This last bit is crucial, as it was not Gorbachev’s intention to turn the USSR into a capitalist state or to see it cease to exist. Most importantly of all, Gorbachev was keen to engage in constructive talks with the United States, and he found a willing partner in Ronald Reagan.
Domestic disasters affected Gorbachev and - in part - forced him to open up to the West. The Chernobyl nuclear disaster that began on April 25, 1986 was the worst civilian nuclear incident in history. Initially led astray by their own subordinates, the Kremlin eventually grasped the gravity of the situation and admitted the scale of the disaster. It also brought the crushing realisation that the USSR could not cope with such a nuclear disaster on its own, undercutting the belief that they might be able to win a nuclear war. Gorbachev also took steps to staunch the bleeding wound that was Afghanistan. He began thinking about how to get out of Afghanistan in 1985, but it took until February 1989 before the last Soviet soldier departed. Gorbachev and his foreign minister Eduard Shevardnadze realised that withdrawal – like American defeat in Vietnam – would be a humiliating loss of face. It would also have domestic consequences: how to explain to bereaved families and mutilated soldiers that it had all been for nothing? It was also in part to do with the United States, who were funding the mujahedeen. Gorbachev wished to establish a regime in Afghanistan – not necessarily a communist one – that could resist what he perceived as the rise of radical Islamism on the USSR’s borders. Washington was less helpful in this that Gorbachev had hoped for, leading to a weaker and more vulnerable post-Soviet Afghan government.
In another post I recounted the story of how Poland and East Germany resisted and then overthrew communist rule. It is important to re-emphasise how different Gorbachev’s approach was and how this was received in the west. His implicit and then explicit renunciation of the Brezhnev Doctrine, his willingness to allow debate, and his openness to new ways of thinking sent a message that this was not the same old same old USSR. Indeed, his inaction when it came to the changes that swept Eastern Europe said more than any explicit statements.
Yet, there were signs that the USSR's leadership could still condone violent repression of pro-democracy movements. December 1986 protests by students in Kazakhstan resulted in 2400 arrests, 450 injuries, and two deaths. In February 1988 Armenian demands that Nagorno-Karabagh become part of Armenia rather than Azerbaijan (both component states of the USSR) were met by violent outbursts against Armenians, while the police and military did nothing to intervene. The Berlin Wall's fall was met with jubilation around the world, and more than a little concern in Moscow. What would happen next?
What did happen was a series of world-historic events that cast a long shadow into the twenty-first century. U.S. president George H W Bush agreed with his West German Chancellor Helmut Kohl that German reunification was in the best interest of all parties. Things moved with remarkable rapidity, with monetary, political, and military reunification being agreed in short order. With the 1990 reunification, the whole of Germany could become part of the European integration project. With the conclusion of the Maastricht Treaty in 1992, the European Community became the European Union, with a united Germany at its heart.
More dramatic (although with the benefit of 20/20 hindsight rather predictable) than the Germany's reunification was the USSR’s collapse. Gorbachev’s willingness to let eastern European communist states go their own way almost inevitably led to movements within the USSR's constituent republics demanding their right to self-determination. Weakened by economic malaise and ever increasing political divisions, the Kremlin could do very little to avoid the USSR's breakup. What is, however, remarkable about this collapse is its speed and its (relatively - although it's important not to over-emphasise the 'peaceful' part) peaceful nature.
First to leave the Soviet orbit were Latvia, Lithuania, and Estonia. This was not without violence. In Lithuania, the moderate nationalist government resigned and communists backed by the Soviet military attempted to take power. 14 Lithuanians were killed and Gorbachev was immediately castigated at home and abroad for a seeming return to the Stalin-Khrushchev-Brezhnev tactics of violent Soviet repression of dissent. In Latvia, pro-Soviet police killed several people, and it briefly seemed that the situation would go completely out of control. After further - mostly futile - attempts to keep the Baltic states on board with economic sanctions, they went their own way and gained independence. This again led to considerable criticism within the USSR of Gorbachev, particularly from hardline communists who castigated him for ‘losing’ Eastern Europe, in much the same way as Harry Truman had been blamed for 'losing' China 45 years previously.
After a failed coup by the hard liners in August 1991, Gorbachev could not salvage any kind of reformed Soviet regime or gain a meaningful agreement for a looser confederation. The Russian leader Boris Yeltsin and his allies (who had played a key role in defeating hardliner the coup) won out, and in November the CPSU was declared illegal and one month later the USSR was superseded by the short-lived Commonwealth of Independent States (CIS). The states that had made up the larger Soviet Union now spun off on their independent trajectories. The Cold War was definitely over, as one side simply ceased to exist.