I’m citing the top answer on this quora question (which itself cites sources), https://www.quora.com/Is-it-true-that-cold-war-spies-would-marry-and-even-have-children-with-their-targets-as-part-of-their-cover, which paints a picture of a U.S. government riddled by Soviet spies. So much so, that Soviet spies successfully infiltrated the Manhattan Project and helped advance the Soviet project by years.
Were the OSS and CIA just novices at human intelligence? Why were the Soviets so able to fill the American government with spies, while American spies in the Soviet Union & Russia can practically be counted on one hand?
In the field of Cold War intelligence historiography, historians indeed generally agree on the Soviet reliance on human intelligence (Humint) as compared to the West's strong reliance on technical collection (Imint, Elint, Sigint). There were a variety of factors at play here and I will describe them briefly.
First of all, the Soviet KGB and GRU could rely on an inherited tradition that started in the 19th century when the Tsar formed the Okhrana. The Okhrana specialized in the recruitment and handling of agents within dissident circles, and by the late 19th century, even operated foreign stations from Paris to infiltrate exiled dissident communities. For instance, Russian Tsarist era dissident Alexander Herzen, while living in London, was spied on by the Okhrana in a way not so different as the KGB did a hundred years later with Alexander Solzhenitsyn. Training manuals of the Okhrana were allegedly used by the KGB School up to the 1980's.
US intelligence lacked this tradition. Although the US Navy developed an intelligence agency in the 1880's, and centralized US military intelligence was formed during the First World War, American politicians tended to disdain and distrust espionage as a method of the ''old world'', belonging to Europe and its totalitarian regimes. Under peacetime, the US vastly downscaled its intelligence capabilities and refrained from such sensitive activities as human spying. The experience of Pearl Harbor changed this attitude among American politicians, who began to realize that wartime spying was useless in the face of a surprise attack - which was considered an accute danger after 1946 when the CIG and its successor CIA were being formed. The CIA was not entirely a club of novices and amateurs - its officers had wartime experience in the OSS and had received training courses from their much more experienced British friends. Nevertheless, the first generation of the CIA still lacked a solid tradition in espionage, and suffered from a sense of adventurism that was useful during war but utterly disastrous during the Cold War.
A second important factor was Communism itself. The Soviet Union simply benefitted from the fact that during the 1920's and 30's, Communism became highly fashionable among Westerners. For many young idealists Communism still offered an alternative and better modern society to the Liberal order, and the rise of Fascism further drove its enemies into the Communist camp. At the same time, few Western countries took counterintelligence seriously, or simply didn't worry much about the Soviets yet. Hence the NKVD and GRU could operate relatively undisturbed in co-opting befriended Communists in Europe, Canada and the US through the Comintern, and use them as talent spotters around universities like Cambridge, where they recruited misguided idealistic students such as Kim Philby to become agents for the Communist cause and apply for jobs in the civil service, parliament, military, foreign office or intelligence services. Many of the Soviet spies surrounding the Manhattan Project in similar fashion felt sympathy for Communism, and could have been uncovered earlier if the British MI5 had bothered to check the names of the confiscated Gestapo archive of known German Communists.
After the crushing of the Prague Spring, which deeply damaged the USSR's reputation and even demoralized the KGB itself, Soviet intelligence began to suffer a serious decline in its Humint capabilities. Hence in the 1970's, Vladimir Kryuchkov, began to circulate directives to all foreign KGB residencies to step up their ''Agent-Operational Work'' and to cultivate more agents.
The third factor is that the KGB benefitted from and exploited the open nature of US and Western societies. One KGB residency abroad, especially in the US and UK, could consist of hundreds of people. Add to that the GRU residencies, and the counterparts of the Warsaw Pact countries and China. For many Western countries, the sheer number of Communist intelligence officers to monitor far exceeded the surveillance capacity of their security services. Hence the KGB, if it wanted to meet with an important agent in the US, would simply send out all its officers from the Embassy in different directions so that the FBI would lack the manpower to follow them all. In Moscow, the situation was the reverse. The KGB, MVD (Internal Affairs/Police) and Communist Party militants had foreigners, and certainly Western diplomats, under constant surveillance wherever they went. In totalitarian systems where even casual contact with a foreigner is suspect, it becomes nearly impossible to foreign intelligence agencies to mix with the locals and cultivate potential targets. Hence the CIA tended to meet its agents such as Kuklinsky (Poland) and Polyakov (Soviet GRU) outside the Soviet Union. Contact was limited to a bare minimum. Such strict measures to protect assets were referred to as ''Moscow Rules''.
It is perhaps interesting to note here that the supposedly legendary Humint capacities of the KGB were virtually powerless against other totalitarian regimes such as Iran, or China. The KGB found it nearly impossible to operate within Iran and the tight surveillance of the Shah's secret police, the SAVAK - trained by the Israeli Mossad. Against China, the KGB was even less succesful. Being completely unable to shake off surveillance from its embassy staff in Beijing, the KGB tried to send Russian Koreans, Kazakhs and Mongolians with false papers across the frontier with China after the Sino-Soviet Split in 1960, but despite perhaps outward similarity to the Chinese, these agents were quickly recognized as aliens by locals who alarmed the Chinese authorities.
So to conclude, the Soviets could build on a long tradition and experience with recruiting and handling human agent networks. The CIA simply lacked this tradition. On top of that, the Soviets made full use of the fact that Communism was very fashionable in the 1920's and 30's, especially among young educated circles, people who could make careers within government. Finally, when Western security services were alerted of Soviet espionage, the Soviets basically overwhelmed them by sheer numbers, making use of the open nature of Western societies to freely roam the country and meeting with locals. In the Soviet Union itself, this was unthinkable for the CIA as any foreigner had KGB gorilla's breathing down their neck.
Recommended Literature:
Haslam, Jonathan. Near and Distant Neighbours: A New History of Soviet Intelligence. Oxford: Oxford University Press, 2015.
Herman, Michael. ‘‘What Difference Did It Make?’’ Intelligence and National Security Vol. 26, No. 6, (2011) pp. 886 – 901.
Garthoff, Raymond L. ‘‘Foreign Intelligence and the Historiography of the Cold War.’’ Journal of Cold War Studies, Vol. 6, No. 2, (2004) pp. 21-56.
Michael Herman, J. Kenneth McDonald and Vojtech Mastny, Did Intelligence Matter in the Cold War? Oslo: Norwegian Institute of Defence Studies, 2006.
Andrew, Christopher & Mitrokhin, Vasili. The Sword and the Shield: The Mitrokhin Archive and the Secret History of the KGB. New York: Basic Books, 1999.
Andrew, Christopher. Defence of the Realm: The Authorized History of MI5. Canada: Penguin Group, 2009.
Haynes, John Earl, Harvey Klehr and Alexander Vassiliev. Spies: The Rise and Fall of the KGB in America. Yale University Press, 2009.