I've heard in passing that a lot of the major events leading up to the dissolution of the USSR/Warsaw Pact (Fall of the Berlin wall, 1991 USSR Coup, etc) caught the Western intelligence agencies by surprise. If that is the case (which it may not be), how did they respond to these sudden shake-ups in their greatest geopolitical foe? I'd actually really like to see primary sources from members of these organizations if they are out there.
If this comes across as a long and kind of meandering answer to the question, I apologise — caveat lector...
Yes, it's true that most of the Western intelligence services were caught off-guard; there's a school of thought that says that Anglo-American intelligence (but particularly the CIA) were so obsessed with their decades-long covert war against the USSR that they blinded themselves to the significance of the Gorbachev agenda, and massively underestimated the momentum of reformist movements in other Warsaw Pact states.
There's also an operational component, however: the fact is that in the mid- to late-1980s, the CIA's network inside the USSR took a bad beating, to the point that they had essentially no meaningful human intelligence (HUMINT) network within the USSR at the time of its collapse. Two particularly well-placed Soviet moles inside the US intelligence community — FBI counterintelligence agent Robert Hanssen, who was a KGB/GRU agent from 1979-2001, and CIA counterintelligence officer Aldrich Ames, a KGB agent from 1985-1994 — helped to compromise a large number of valuable intelligence assets inside the USSR. The net result is that the Western intelligence services are often lampooned — not without justification — as having watched the collapse of the USSR on CNN.
The end of the Cold War prompted a difficult period of introspection — even an all-out identity crisis — for the Western intelligence services, which had accrued enormous budgets and vast resources to fight an enemy that no longer existed. Yet they had also been watching the rise of new threats during the 1970s and 1980s that became crucial strategic priorities in the 1990s: the biggest ramification of the end of the Cold War was that counterintelligence declined in importance, and was to some degree superseded by counterterrorism, counter-proliferation and other emergent threats.
(Note that post-Soviet Russia was, and remains, a significant intelligence threat and target — none of the Western agencies have stopped spying on Russia, and nor has Russia stopped spying on the West. The point is that it's no longer the premiere global threat.)
Tim Weiner's Legacy of Ashes, which is the best/most comprehensive (and also the most sharply critical) history of the CIA I've encountered yet, contains a rather elegant analysis of the late-Cold War failures and early-1990s ennui within the Agency.
Weiner also quotes Milton Bearden, head of the CIA's Soviet desk in the late '80s and early '90s, observing that the end of the Cold War precipitated a brain drain of capable intelligence officers leaving the Agency:
The loss of the Soviets tore out the CIA's heart. How could the agency live without its enemy? "It was easy, once upon a time, for the CIA to be unique and mystical," Milt Bearden said. "It was not an institution. It was a mission. And the mission was a crusade. Then you took the Soviet Union away from us and there wasn't anything else. We don't have a history. We don't have a hero. Even our medals are secret. And now the mission is over..."
[...]
"The ultimate tragedy is spiritual," he said. "Most of the younger officers I knew have resigned. These were the best and the brightest. Eighty or ninety percent of the people I knew, halfway through their careers, have packed it in. There was very little motivation left. The enthusiasm was gone. When I joined the agency, back in '76, there was a tribalism. The esprit de corps that the agency had was created by this tribalism, and it served a good purpose." And now it was gone, and most of the clandestine service was gone with it.
In November 1991, the George H. W. Bush administration circulated 'National Security Review 29', a memo which captures some of that sense of institutional listlessness and the need for a fundamental change of direction in the US Intelligence Community. Mostly drafted by then-Director of Central Intelligence (later Defense Secretary) Robert Gates, the memo read, in part:
CIA and the Intelligence Community have their roots deep in a Cold War that is now over and the threat of a Soviet military that is now struggling at home to preserve its very existence, to avoid its breakup into multiple republic armies. We are in transition from watching Soviet operational readiness to wondering about the control of Soviet nuclear weapons. The Soviet Union and Eastern Europe have been transformed, the Warsaw Pact dissolved, and Soviet activism abroad (especially in the Third World) dramatically curtailed. There is growing interest here at home in our intelligence services tackling new issues and problems. Together, these developments require a top to bottom examination of the mission, role and priorities of the Intelligence Community.
The British intelligence community followed a similar strategic path in the 1990s — counterterrorism became a higher priority, and the counterintelligence function declined. One salient difference, though, is that CT wasn't an especially 'new' threat for the Security Service (MI5), which had 20 years of combating the threat of the IRA under its belt by the time the Cold War ended.
But the reallocation of resources from counterintelligence to counterterrorism has been dramatic: historian Christopher Andrew points out that by 2007/08, MI5 "spent a mere 3.5 per cent [source] of its budget on counter-espionage." Intelligence budgets were cut back significantly in the 1990s, and only really began to rise again after 9/11.
I haven't got the time and space to dig deeper into the British side of things right now, but I'd suggest looking at the 1995 annual report of the Intelligence and Security Committee, the committee created in 1994 to provide for the first time parliamentary oversight of the British intelligence services. That document is their first ever report, and provides an interesting perspective on what British intelligence policy looked like in the mid-1990s, and how the intelligence services were adapting to the new world in which they found themselves. It is, however, a parliamentary document — so it's not the most riveting read...
Now, a long aside about sources in intelligence history...
Just a general point about primary sources in this field: they're often hard to come by in intelligence history, for the obvious reason that it's a discipline focused on clandestine activities.
Documentary/archival evidence usually only emerges either after declassification — which is generally only done after a significant amount of time has passed, and only for material deemed non-prejudicial to state interests or ongoing operations (which means a lot of the really juicy stuff may never see the light of day) — or by being leaked, which is more rare, especially at significant scale, but does happen; the Edward Snowden leaks, for example, or Vasili Mitrokhin's leaking of a huge quantity of KGB archival material. However they emerge, that archival evidence is the primary foundation for the study of intelligence.
Historians are occasionally given broader/deeper access to intelligence services' archives, but this is relatively rare. All three UK intelligence agencies have recently allowed authorised histories to be published by professional historians, but their scope is often curbed by the agencies themselves:
To my knowledge, none of the US intelligence agencies have ever allowed the kind of archival access that would enable that kind of comprehensive organisational and operational history to be written (though some interesting work has emerged from the Agency's Center for the Study of Intelligence.)
For the rest of our understanding of intelligence issues, we tend to rely on two other sources: investigative journalists, who in this field have frequently performed a useful role as the 'first drafters of history', especially as regards the US Intelligence Community — reporters like James Bamford and Tim Weiner have produced some remarkable work.
The last genre is the memoirs or other publications from former intelligence officers (which can make for fascinating reading, but can also muddy the waters further — Peter Wright's widely-read Spycatcher is my favourite example of this latter problem; it made a huge impact, but it's also fatally flawed in some respects.)
While I think many observers were shocked by the collapse of the Soviet Union considering it appeared very strong in the late 1970s, the CIA, especially the Directorate of Intelligence, knew Gorbachev faced greater challenges than he could handle in the late 1980s.
Once Gorbachev came to power in 1985, the CIA believed he would have to undertake sweeping reforms fast in order to arrest the Soviet Union's structural economic problems. His reforms, however, were often too limited between 1985-1987 because he had to amass his own political power and prevent conservatives in the Soviet leadership from challenging him. Once he had that power, the CIA thought it was not too late to turn the Soviet Union around BUT that doing so would be increasingly unlikely. The CIA did not think a collapse was likely in 1987-88 but that the Soviet Union would turn more and more into a hollow shell unable to compete with the West in military, economic, and political areas.
By November 1990, a National Intelligence Estimate characterized the situation inside the Soviet Union as one likely heading towards "deterioration short of anarchy" with Gorbachev's power continuing to decline and the prospects of viable political and economic reforms unlikely. A few months later in May 1991, the CIA considered Gorbachev done for. As another report found:
Gorbachev remains an important player on the Soviet political scene, especially in foreign and defense policy, but his domination of It has ended and will not be restored
In short, the Soviet Union is now in a revolutionary situation in the sense that it is in a transition from the old order to an as yet undefined new order. Although the transition might occur p'eaceably, the current center-dominated political system is doomed. As happened in Eastern Europe over the past two years, the ingredients are now present in the USSR that could lead not only to change in the regime, but in the political system as well.
In the midst of this growing crisis, CIA analysts considered the likelihood of a coup against Gorbachev increasing, and once it happened in August, they didn't expect the Soviet Union to limp on much longer.
The two estimates I mentioned are found here and here.
I encourage you and anyone interested to check out the CIA documents available in the CIA's FOIA Reading Room Reagan Collection. It contains over 200 CIA documents on US-Soviet relations, detailing in particular the declining Soviet economy, political power struggles, and the prospects of the Soviet leadership overcoming their system's mounting problems.