So: there are many stories about British counter-intelligence during WW2, and their deceptions of the Nazi's. They indicate a intelligence agency which was on top form and good at their jobs. However, during the Cold War (even from the early Cold War), you get alot of stories of how compromised British intelligence is: The Cambridge Five are the keynote. So why is there this discrepancy? Or is my perception that there is a discrepancy wrong?
The Cambridge Five are a tricky example, because they weren't really a phenomenon of the Cold War: all five were recruited in the 1930s, and were at the height of their operational effectiveness during the Second World War. The entire network was effectively neutralised by the end of 1951.
Anyway, the success or failure of British counterintelligence really has as much to do with the capabilities of their opponents as with MI5's competence. In short: the Security Service was effective in countering German intelligence because the Germans were simply not very good at building and running productive agent networks abroad. The Abwehr, Germany's military intelligence service, spent much of its time engaged in a vicious turf war with their SS counterparts, the Sicherheitsdienst (SD). There was essentially no effective coordination or collaboration, and little institutional experience in running complex foreign intelligence operations. (It should be said, however, that the German counterintelligence and internal security services were far more capable.) The Double-Cross System was a masterpiece of disinformation and deception — but it also owed much of its success to a healthy dose of luck and the inability of German intelligence to corroborate and cross-check the information they were receiving.
In the Soviet intelligence services, however, MI5 faced a far more dangerous foe. The NKVD, which eventually evolved into the KGB, proved substantially more effective than the German services in recruiting willing agents overseas; among other things, they had an international recruitment network at their fingertips, in the form of the Comintern and talent scouts in national communist parties and movements around the world.
The real reason that Britain and America were so vulnerable to Soviet infiltration in the 1930s and 1940s, however, is because they simply weren't watching out for it. Britain was far more concerned with Germany, and both countries were woefully blasé about Soviet intelligence capabilities. From the mid-1920s until at least 1936, Britain's Secret Intelligence Service didn't even maintain a station in Moscow.
The defection of NKVD officer Walter Krivitsky to the United States in 1938 alerted the Western intelligence services to the threat (Krivitsky was extensively debriefed by the FBI and MI5, and even published a book), but they continued to underestimate the scale of Soviet espionage for several more years. Only after the defections of American agents Whittaker Chambers and Elizabeth Bentley, and the success of the VENONA cryptography programme, did the magnitude of Soviet infiltration begin to become clear to policymakers in the West.
It's true that the discovery that Soviet agents had penetrated the uppermost levels of the British government and intelligence establishments was a body-blow — it took years for the Anglo-American intelligence relationship to recover, particularly because of Kim Philby's close collaboration with US intelligence — but the perception that British intelligence was on the back-foot throughout the Cold War is a little unfair. The Soviets recruited few British agents of the Cambridge Five's calibre after 1950; perhaps only SIS officer George Blake and Ministry of Aviation bureaucrat Frank Bossard really compare.
It's true that over time the UK became a lower strategic priority than the US; the Soviets were rather more concerned with the rising superpower than the declining empire. But Britain was nonetheless an active participant in the intelligence war, and both MI5 and MI6 achieved some critical successes. The most notable counterintelligence coup was certainly Operation FOOT, the simultaneous expulsion in 1971 of 105 Soviet intelligence officers under diplomatic cover, which effectively crippled Soviet intelligence-gathering in the UK for several years.
Some good sources for more information: