Before we discuss the history of economic thought, it's generally necessary to point out that Keynesians, Neoclassicists, and Monetarists are all intraparadigm schools of thought within the paradigm of market-based capitalism, meaning that they generally all prefer a generally free market, but differ on their emphasis on governmental policies.

From the 1930s to the 1970s, Keynesian Thought strongly pervaded governmental policy-making. Its core tenets were wrapped around the notion of Demand-Side Economics, in which Consumer Demand is fundamentally the driving force of the economy, in contrast to what was previously assumed as Say's Law, or the notion that Supply Creates its own Demand. Keynes was notable most famously for advocating for government intervention, but also, lesser known for advocating the notion of underconsumption, as opposed to overproduction. Underconsumption suggested that firms will fail to meet equilibrium prices, usually through a major shock in aggregate demand, such as a recession. Thus, it might be necessary for the government to step in.

One of the proposed solutions to guide policy is the Philips Curve, which suggests an inverse relationship between inflation and unemployment. Essentially, as inflation goes up, unemployment goes down, largely because it becomes cheaper for firms with "sticky wages" or a rock bottom, such as a minimum wage, to hire more laborers as the nominal wage remains the same, but the purchasing power goes down. Thus, more people are employed and can consume more.

The Achilles' heel of this analysis is that observations were only made in the short-run, not the long-run. To be effective, the Philips Curve assumes that there is an inverse relationship between unemployment and inflation all things held equal. The problem with it was that it was later observed that inflation not only effects the labor market, but it also effects many other markets that use the labor market as an input. So as relative purchasing power declines, so does aggregate demand.

Additionally, Milton Friedman, who claimed much more prominence with the award of Nobel Prize in Economics for proving that the monetary contraction in 1929 caused the Great Depression, also theorized that in the long-run, both workers and firms will take inflation into account and will simply adjust for inflation as they expect it to occur. Thus, you get all of the negatives, some even worse, without any of the positives.

In addition to this policy failure of Keynesianism, you also had the Arab Oil Embargo, which seriously damaged Aggregate Supply, as oil prices shot up and there were chronic shortages, the Nixon Shock, which implemented price and wage controls, which directly interfered with the economy's flexibility, the rise in Globalization specifically in East Asia, with Japan and China, but also in Europe and Parts of Latin America. These forced previously uncompetitive industries, such as the Automobile and Steel Industries into major declines. Steel production in the United States ultimately completely shut down, and is a major factor in the depression affecting the Rust Belt of the Northeast. Lastly, many observers blamed excessive regulations for interfering with business practices.

These issues lead to a paradigm shift to place more emphasis on reducing government involvement, mostly in the form of subsidies to prop up ailing industries, as well as the regulatory infrastructure that was deemed more harmful than helpful, as well as generally bad policies, such as an extremely loose monetary regimen.

Mainstream economics prior to the late 1970s believed in the long term success of the Phillips Curve, which held that high inflation would correlate with low unemployment, and wasn't inherently a bad thing.

There were voices opposed, most notably by free-marketists, who did predict eventual periods of both high inflation and unemployment, but they remained on the fringe because of the success of Keynesian economics in the post war period. Once this system began to falter, previous "fringe" positions started to become more and more legitimised.

To put it simply, economists didn't predict stagflation because their paradigm/ideology/perspective, whatever you want to call it, didn't predict it. Once it occurred, a previously fringe paradigm/ideology/perspective began to grow in academic and political prominence.