We can't really answer why things didn't happen, other than to say, well, it was not really in anyone's interest to start World War III.

In the case of the Korean Air Lines 007 shoot-down, it was clear to the US analysts that a) this was probably a mistake, and b) that the best thing the US could do with this mistake was to try to capitalize on it politically. So the response was a heavy diplomatic campaign meant to embarrass the Soviets (who didn't even really want to admit it happened) and to try and take a moral high ground with both domestic and international audiences.

(Whether the US had as much moral high-ground to claim is of course part of what is debated today — the US arguably played a major role in creating the conditions for the shoot-down with their aggressive exercises in violating Soviet airspace in the region as a means of testing out their air defenses, which put the Soviets deeply on-edge.)

Now I am not 100% sure I would say "cooler heads prevailed," and while this is a "diplomatic" approach (as opposed to a "military" one), it was still pretty aggressive. The Soviets took it as a sign of American bad-faith, and it was interpreted as further evidence that the US was trying to set up the conditions for a massive first-strike attack against the USSR. The US was not, in fact, trying to do this, but that is sort of beyond the point when you are talking about perception. So it was still a fairly aggressive response, and one that was deliberately so. Diplomacy can be aggressive too, although it tends to be less escalatory than actual military operations!

One can imagine many worlds in which the Cold War went "hot" (or at least "hotter" than it was, as it was not exactly frigid), in which the decisions went one way or the other. You can imagine a scenario in which the US decided to pursue this issue a different way, or in which the Soviets interpreted the US response another way, and things went in even more dangerous directions than they actually did. To look for a single answer as to why it didn't at any tense moment is probably fruitless at best, and misleading at worse, because it assumes that there was a sort of rationality and control at work. The people who were involved in these moments tended to emphasize that quite a lot of what got everyone through the Cold War was "luck." We can see this as a shorthand way to say, "there wasn't much of any reason that we had control over" — luck just means the absence of control, which means that there isn't a necessary, logical, rational answer. (On a serious study of the role of "luck" in history and the Cold War in general, see my article here with a colleague who has spent a lot of time thinking about this issue.)

The most "rational" answer you can offer up is that all of the sides involved understood that there would be huge costs to ultimate escalation, and so worked to avoid that. This is the "deterrence" answer. But it only takes you so far. The US approach to the Soviet Union in 1983 was, to a degree, deliberately escalatory — but always stopped short of what the US thought would be "too risky." But the US determination of what is "too risky" is not necessarily the same as the Soviet one, and therein lies the possibility for severe miscalculation or error, if one side crosses a "red line" of the other without realizing it. The KAL 007 shoot-down itself was an example of just such a miscalculation and error. The best you can say about the aftermath is that the US response pursued a path that it thought would get it what it wanted at a minimal or acceptable cost, and their judgment on that was (to whatever degree of accuracy) good-enough that it didn't break out into war.

The best overall book I know of on the crises of 1983, with a lot of discussion of KAL 007 and its aftermath (and Able Archer 83, and the Stanislav Petrov incident, and the Pershing II crises, and many other things of that dangerous year) is David Hoffman's The Dead Hand.

Relatedly, I am a big fan of this comic by Zach Weinersmith (SMBC), which espouses the "Anthroponuclear Multiple Worlds Theory," and frequently bring it up when talking to people about the dangers of the Cold War. It's an amusement, of course, but the framework of parallel universes is a way to highlight just how contingent things were in this period — that the survival of the world should not in any serious way be taken for granted.

September 1, 1939 marks the day Hitler invaded Poland and the official start of WW2.

In a different universe, September 1, 1983 could have marked the start of WW3.

I know that some of our critics have sounded off that somehow we haven't exacted enough vengeance. Well, vengeance isn't the name of the game. Short of going to war, what would they have us do?

-- Ronald Reagan, on a call to the Republican Western Regional Conference

Relations did tank due to the incident. Gallup polls showed after that public opinions of Soviets were the worst since 1956 (that'd be the Hungarian Revolution when the tanks rolled in). Had the incident happened at a worse moment or especially with a more trigger-happy leader at the helm (say, Goldwater circa the 1960s) things could have turned into a shooting war.

...

On that fateful September day, Korean Airlines Flight 007 (New York to Anchorage to Seoul) sent a message at 3:23 AM, stating it was at the eastern tip of Hokkaido, Japan.

It was not. It was more than a hundred miles off course, due to pilot error in regards to setting the autopilot.

An initial message from Korean Airline claimed it had been forced to land in Soviet territory by their air force. This was entirely untrue, and the Soviets were quick to deny it. As reported on Tokyo radio:

When Togo [at the Japanese embassy in Russia] questioned the official [from the Soviet Foreign Ministry] whether this meant the missing South Korean airliner was not in Soviet territory, the official replied that it was not within Soviet territory since it had not landed on Soviet territory, Togo told Japanese newsmen. Japanese Embassy sources said later they could not rule out the possibility the ill-fated South Korean plane was shot down by the Soviet Union.

This was the first mention of the possibility of the plane being shot down, something later confirmed about 24 hours after the plane's disappearance by TASS in an official statement:

An unidentified plane entered the airspace of the Soviet Union over the Kamchatka Peninsula from the direction of the Pacific Ocean and then for the second time violated the airspace of the USSR over Sakhalin Island on the night from August 31 to September 1. The plane did not have lights, did not respond to queries and did not enter into radio contact.

Fighters had been scrambled and shot the plane down. (The lights part is untrue; Major Gennadiy Osipovich, the one who actually launched the missiles, later stated this was one of the lies that cropped up about the incident. Part of the confusion came from the people involved lying to Moscow in order to avoid punishment.)

The Soviet foreign ministry was initially in the dark as everyone else, although before TASS statement came out the Secretary of State of the US (George Shultz) did receive notice with the information from TASS and a "possible crash".

Shultz held a press conference the next day:

At 1821 hours the Korean aircraft was reported by the Soviet pilot at 10,000 meters. At 1826 hours the Soviet pilot reported that he fired a missile and the target was destroyed. At 1830 hours the Korean aircraft was reported by radar at 5,000 meters. At 1838 hours the Korean plane disappeared from the radar screen.

adding that "We can see no excuse whatsoever for this appalling act."

Keep in mind, for this very moment, there was at least some leaning to a fix in Soviet-US relations. National Security Decision Directive Number 42 had been declared by Reagan the year before, calling for "international cooperative activities" in space (meaning with Russia). A new de-escalation arms deal was still in the works, and on the very same day as the KAL 007 incident the Agriculture Department announced they had made their first sales of grain to the USSR in a five-year agreement.

The grain agreement announcement was unfortunate, and there was some concern of sanctions (the traders in Chicago did a big sell-off) but despite the demand of sanctions from some conservatives there was no move to call-off the deal from the President.

The President's initial response was to give on September 5, a 16 minute speech (which you can watch here):

This crime against humanity must never be forgotten, here or throughout the world.

Reagan noted, importantly, that this was not the first such incident: KAL 902 had been shot down in a similar manner in 1978, and there was grave concern for the general safety of air travel altogether. His very specific demand is that

They owe the world an apology and an offer to join the rest of the world in working out a system to protect against this ever happening again.

Even though he called it a "terroristic" incident Reagan kept to the same position. Later, Reagan reiterated in an interview with TIME:

Obviously you are tempted to to think about vengeance, but there is no way you can avenge such a thing .... But what you have to look for is what you can do, first of all, to get restitution for the families of the victims, and what you can do to see that this never happens again.

Echoing a similar sentiment, in a service attended by Reagan for the dead at the National Cathedral the Bishop John T. Walker said:

We cannot accept that the people of the Soviet Union are inherently immoral. Rather, we must believe that the context for this action is suspicion, distrust and fear.

Despite conservatives calling for a much stronger backlash (George Will called Reagan's response "pathetic") this was the general attitude of the government, and while the US had some explicit claims that the Soviets knew they were shooting at a civilian aircraft, there was the diplomatic opening to consider the event an accident.

There was still some diplomatic kerfuffles to be had (including a UN resolution shot down by the USSR's veto), but at the time the ending was essentially a stalemate: the Soviets claimed they shot down a spy plane and would not budge or talk of reparations. 1985 is when things turned around with Gorbachev entering power and (perhaps even more importantly) Shevardnadze becoming head of the Foreign Ministry a few months later. This is when relations started to warm up again, and the airline incident was temporarily put under the rug as more important talks went forward. For example, the Intermediate-Range Nuclear Forces (INF) Treaty happened two years later, the first agreement that led to a reduction of nuclear arsenals.

In December of 1990 Shevardnadze gave an official apology to South Korea for the incident and resigned his post.

...

Dallin, A. (2022). Black Box: KAL 007 and the Superpowers. United States: University of California Press.