Second attempt at this.
I've just finished reading 'Skunk Works' by Ben Rich about Lockheed Martin's special project division (great book, highly recommended).
During the section on the development of U2 spy plane, Rich mentions how spy planes that flew at a lower altitude, prior to the introduction of the U2, were being shot down by Soviet MIGs, to the point that they introduced fighter escorts, which then engaged in dogfights with the MIGs with both sides suffering casualties. Rich mentions how the waste of the lives of young American pilots was a driving factor at Skunk Works for the success of the U2.
Now I think I'm fairly aware of the basic 'semi direct' engagements of the Cold War (Soviet pilots in Korea and Vietnam, Bay of Pigs, CIA in Afghanistan etc.) but all of these had an aura of deniability when it came to NATO and Warsaw Pact forces directly engaging each other.
But with something like the downing of Gary Powers or the death of Major Arthur Nicholson in the DDR, it seems it was a massive deal at the time and was widely known to the public.
So did these pre-U2 engagements really happen and if so, how did they, if you'll pardon the pun, fly under the radar with neither side publicly acknowledging them? And were there other similar events that were also not public knowledge?
Frank Rich was correct in that there was a considerable Anglo-American attempt to penetrate the territorial airspace of the USSR with reconnaissance aircraft. These aircraft ranged from converted WWII aircraft like the RB-29/RB-50 (reconnaissance versions of the B-29) and converted jet bombers like the RB-45, RB-47, and RB-57. The performance envelope of these aircraft was quite high, but Soviet ground and air defenses were increasingly able to intercept them. Although not all of these missions have been declassified, William Burrows estimates that sixteen of these aircraft were lost on these missions, with a majority of them lost from Soviet action.
These shootdowns were diplomatic incidents at the time. The Soviets brought up the 1 July 1960 shootdown of an RB-47 at a session of the UN protesting American penetration of the Kola Peninsula. For its part the State Department demanded that the USSR return the two surviving aircrew and asserted that the shootdown occurred in international airspace. The 1960 RB-47 incident was of a piece with earlier aerial clashes that took place. If the Soviets managed to damage or shootdown these snoopers, they would make a big stink over American provocations and sabre-rattling. The US would in turn deny any provocation and that the aircraft were engaged in peaceful activities in neutral airspace or inadvertently strayed into Soviet territory.
The American diplomatic defense of these incursions were often lies and dissimulation. SAC and other military agencies authorized these intrusions as a means to both ascertain Soviet air defenses and find out the disposition of Soviet military installations at the edges of the USSR. One of the reason why these incursions did not reach the level of the Powers incident was timing. The height of these earlier air battles was during the first half of the 1950s. For example, three RAF-crewed RB-45s in 1952 penetrated the GDR, the Baltics, and European Russia in a night reconnaissance operation which would be repeated two years later, albeit this flight had to turn away from Kiev because of Soviet responses. The USAF likewise conducted similar overflight operations in which small-scale formations or single ships would make a dash over Soviet strategic bases. These provoked a considerable Soviet response such as in 8 May 1954 when an RB-47 overflight of the Kola Peninsula provoked a considerable air battle in which MiG-17s tried and failed to shootdown the American plane. As foolish as this might seem in hindsight, part of the reason this was countenanced was that this was arguably the point at which the Cold War was at its most frigid. These overflights occurred contemporaneously with the Korean War and while they were less aggressive than the overflights of the PRC and DPRK, they were still caused a good deal of provocation. One of the reasons Eisenhower backed away from supporting the military overflights in his second term is that he noted that the American public would be apoplectic if the Soviets engaged in similar overflights of US territory. But such reconnaissance flights were a calculated risk on the part of SAC as it allowed them to ascertain Soviet strategic potential and air defense response times when these were open questions. The Soviet's nuclear capabilities were also unknown at this time as were their actual bases. The Korean War had a wide-ranging impact on the American intelligence and defense establishment not to be caught unawares again so it encouraged a more aggressive approach to aerial reconnaissance. While the Truman administration was reluctant to authorize overflights (although SAC could ignore this), the more hardline Eisenhower administration during its first term had little such qualms. The Korean conflict also helped dampen any American domestic counter-reaction to these incursions and their losses.
It also helped matters that most of the incidents occurred at the edges of Soviet power. The 1952 RB-45 overflights and their successors were the exceptions as most aerial flights "tickled the Soviet bear" by overflying the Soviet periphery. This was a reflection not only of the limitations of these reconnaissance aircraft to survive over Soviet airspace for a long period of time, but also the nature of the Soviet threat. ICBMs would increasingly take the center of the Soviet nuclear arsenal in the mid-1950s, but they were few in number until the following decade. The main nuclear threat to the US in this period was the manned bomber and most Soviet bombers of this period would have to be based at the edges of the USSR like Kamchatka to reach the lower-48 states. These polar incursions gave American officials a degree of plausible deniability as they could argue that navigation was difficult in this region of the globe- which was true- and that the Soviets overreacted to innocent mistakes. Other incidents occurred over territory that the US did not fully-recognize as proper Soviet territory such as the Kurile Islands or Sakhalin.
The Powers shootdown was a very different matter than these earlier overflights. For one thing, the Skunkworks' plane was designed from the outset to make a deep overflight of the USSR. The US cover stories had a much thinner veneer of veracity when the plane was shot down over Sverdlovsk, which was considerably inland. Lockheed and the CIA's self-destruct mechanisms proved to be ill-suited to the job, so they it was inescapable to deny that this was a spy plane. Moreover unlike earlier incidents, this one took place in a period of deescalating US-Soviet tensions. The shootdown precipitated a halt to this particular thaw in the Cold War, which made it a diplomatic embarrassment or the US. Khrushchev made such a big deal out of the U-2 overflights during the Paris summit that even some of the Soviet Premier's entourage thought their leader was making too big of a deal out of the incident.
This shootdown did not fundamentally change the patterns of aerial reconnaissance of the USSR outside of direct overflights. But aerial reconnaissance methods grew more sophisticated so that riskier incursions were not as necessary to gain electronics and signals intelligence. Nonetheless, American snoopers would still patrol the edges of Soviet airspace through the remainder of the Cold War and periodically play games of chicken to prompt a response among the air defenses. This was a contributor to the KAL 007 Sakhalin shootdown in 1983 in which American RC-135 ELINT aircraft had a habit of running to the edge of Soviet airspace in Kamchatka and turning away at the last minute. This engendered a Soviet hairtrigger response in the local air defense network that was primed to capitalize on any American mistakes. This ensured that when the KAL 007 mistakenly entered USSR airspace, the local defense network was ill-equipped to put the brakes on the interception despite some indications that this was civilian flight.
There were other aerial incidents during the Cold War unrelated to aerial reconnaissance. Navigational errors possibly contributed to the so-called Air battle over Merklín in which Czechoslovakian MiG-15s one of two F-84s that had strayed over the border. On 12 March 1953 Soviet MiG-15s shotdown an RAF Avro Lincoln in the Berlin air corridor after it had strayed from its flight path. There were a host of other shootdowns and forced landings in which American or American-allied aircraft strayed on the wrong side of the border and got caught. For their part, the Soviets did not maintain as an extensive aerial reconnaissance operation. Some TU-16s did overfly Alaska, which prompted the beefing up of Alaska's air defense network. MiG-25 reconnaissance variants performed as a role analogous to the SR-71 in that the Soviet plane could fly at high altitude to gain information without directly violating airspace. The Foxbat replaced the Yak-25RV in the role of aerial intelligence and it performed high-profile missions with Soviet pilots, especially during the Yom Kippur War. But the Soviets overall did not place as high a priority on strategic aerial reconnaissance as NATO or the US, preferring satellites or human intelligence sources to gain data on NATO facilities and capabilities.
The Cold War aerial incidents were not some covert operations completely outside of the public's eye. Us disinformation and obfuscation contributed to a degree of confusion about the exact nature of these incidents, but the Soviets did actively publicize them and their results. These shootdowns struck a minor note during a period of quite chilly Cold War tensions. The most dramatic aerial battles such as the 1953 Merklín incident occurred when the US was fighting an actual proxy war in Korea. Later incidents such as the Powers and 1960 RB-47 shootdowns occurred during periods of relative quiet in the Cold War so they loomed more dramatically in the news and subsequent collective memory.
Sources
Burrows, William E. By Any Means Necessary: America's Secret Air War in the Cold War. New York: Farrar, Straus and Giroux, 2001.
Laurie, Clayton D. and R. Cargill Hall. Early Cold War Overflights, 1950-1956: Symposium Proceedings. Washington, DC: Office of the Historian, National Reconnaissance Office , 2003.
Peebles, Curtis. Shadow Flights: America's Secret Air War against the Soviet Union. Novato, Calif: Presido, 2002.