It often seems that in the 1960's and 70's the middle east was very similar to western countries. Lebenon and Iran in particular appeared progressive, open and western.

This is a huge and loaded question.

First, I would heavily caution with saying that countries in the Middle East were very similar to western countries in the 60's and 70's. The Middle East has an extremely diverse set of governments and countries varied considerably in their policies and even in the styles of governance. Indeed, by the 60s and certainly by the 70s, oppressive states in Syria, Iraq, and Libya were established - in fact, most of the nations that had revolutions/civil wars during the Arab Spring can trace their government's roots to the 60's and 70's.

Even your assertion with Lebanon and Iran appearing progressive, open and western must be tempered with the fact that much of our media focus on Lebanon and Iran during that era, and pictures available from that era, were generally taken from the urban population (who had access to said cameras), often the elite.

For instance, Iran today has a rural population of approximately 21 million people. Compare that to the numbers in 1980, with around 19-20 million people, and then contrast that with the fact that Iran's population in 1980 was around 40 million. Indeed, in 1950, Iran's urban population was 27 percent while in 2005 it was up to 67%, and from that you'll see that in the 60's and 70's most of Iran was actually quite rural and far from the pictures of the elite in places like Tehran.

What changed in the middle east and caused the current chaos?

There are a lot of theories and thoughts into what changed in the middle east.

One area to remember is that Islamist militants aren't a new thing. Indeed, the founder of Saudi Arabia, King Abdulaziz of House Saud, forged an alliance with the Ikhwan - fierce bedouin warriors from the center of the Arabian peninsula, and followers of the strict Wahhabi form of Islam. In fact, after the establishment of Saudi Arabia in the 1920s, King Abdulaziz had to govern the country and deal with the Ikhwan who had started to rebel and disobey his orders not to attack British holdings in Transjordan and Iraq. Furthermore, the Ikhwan blame King Abdulaziz's western-styled modernization as a betrayal of fundamental Islamic teachings and a reason to rebel against their king - the Ikhwan were crushed by King Abdulaziz, however, and their status as an effective fighting force ended by the early 1930s.

The post-WW2 era in the Middle East was dominated by the on-going Arab-Israeli conflict. Here, militant Islam took a backseat if you will to the nation-state conflicts going on between Israel and Egypt, Syria and their allies. In a lot of ways, while Egypt, Syria and their allies were able to field tens of thousands of men in organized warfare against Israel, militant Islamists just weren't at the forefront of any of the conflicts going on.

However, their modern roots can be traced to a lot of these governments at war with Egypt. In 1952, Gamal Abdel Nasser helped organize the revolution that overthrew King Farouk of Egypt. Nasser was part of the Free Officer's Movement and he rose to become President of Egypt. He was a believer in pan-Arabism, which sought to unite the Arab world as a single nation, which had been divided up by European powers after WWI and after the collapse of the Ottoman Empire.

With the collapse of the Ottoman Empire, the caliphate had essentially been disestablished. The caliph was both the religious and political leader of Islam, and in this post-caliphate world, the questions asked included under what basis could the Arab world be united? To do so, Nasser - and many followers of pan-Arabism - sought to create a state united not by religion but by a common culture and language.

As thus, the revolutionaries in Egypt sought to create a secular constitution as the basis for the new government. In 1954, an assassination attempt on Nasser by a Muslim Brotherhood member prompted the Egyptian government to crack down heavily on the Muslim Brotherhood, which has always fused religion and politics. They would be outlawed as an organization for nearly six decades until they resurfaced in the Arab Spring.

Nasser's leadership in the Arab world and his pan-Arabism ideals spread. Boosted with his leadership of Egypt, the primary foe of Israel - a foe every Arab could agree upon - Nasser was quite a popular figure and inspired many new generations of revolutionaries across the Arab world. In fact, the rise of the Baath party (which espouses pan-Arabism and Arab nationalism) in both Syria and Iraq, as well as the rise of Gaddafi in Libya, all shared many of Nasser's thoughts. And, since many of these governments ruled in areas with large religious divides, forced secularism in government and daily life was advantageous for governance.

As thus, this is why you often see from newsreels and photos of the 60s and 70s in the Arab world the "secular" look of people, government, and lifestyle.

---

Part 2 in reply

A lot of it was because of regional instability. There are many factors going into this, and it varied greatly by country. For example, Egypt under Gamal Abdel Nasser in the mid-20th century was extremely hostile to Islamist groups like the Muslim Brotherhood. And Saudi Arabia's Sunni Wahhabist ruling regime is what is usually thought of when people say "fundamentalist Islam."

Because of things like this, I would argue that focusing on the religious aspect is too narrow in scope. The situation has long been heavily related to regional politics, ruling regimes, socioeconomics, and nationalism. Egypt, Saudi Arabia, Iraq, and Iran over the twentieth century all went to great lengths to expanding their regional influence. All of this along with the political instability of certain states allowed a breeding ground for radical groups to rise.

Iran is an interesting example. They are Twelver Shi'ites, one sect within Shi'ite Islam as a whole and very different from non-state terrorist groups. The Shah's collapse was more than just a victory for what one might call "militant Islam;" it was a major political event. Prior to 1979, the United States followed the "twin-pillar policy" in which they relied on allies Iran and Saudi Arabia as bulwarks to prevent the spread of Soviet influence in the Middle East. But the Shah ran a repressive regime, and there was hope that the revolution would bring about positive reform in the country.

It was a huge development, and neighboring regimes feared a revolutionary wave (similar to the Arab Spring). From 1979 onward, a balance of power situation developed between Iran, Saudi Arabia, and Iraq. All wanted to expand their influence in the region, and all had very different ideologies. Iran challenged Saudi Arabia's claim as the leader of the Muslim world, and around that time Saddam Hussein, feeling threatened, invaded Iran. As I said, both he and the Saudis were concerned that the Iranian Revolution would inspire revolutionary waves across the region. Iraq's population is majority Shi'ite, and the Sunni minority ruled the country under Saddam. And while only about ten percent of the Saudi population is Shi'ite, they are large concentrated in the Saudi Eastern Province, a major oil producer. During the Iran-Iraq War, the Saudis at various points backed Hussein financially, and this earned ire from the Iranians.

When Saddam invaded Kuwait, the Saudis and Iranians were pushed closer together, as they both desperately wanted to prevent Iraq from annexing what Iraq called its "19th province." This ultimately led to US involvement in the Gulf War and American troops in Saudi Arabia, an event which radical groups used to declare the illegitimacy of the Saudi dynasty.

The Saudi Arabia-Iran-Iraq situation is just one example, but it is indicative of the environment in which extremist groups--whether inspired by nationalism, religious fundamentalism, the desire for power, or a mix of these--arose. The were parallel conflicts throughout the twentieth century as Middle East countries emerged from colonialism.

Selected sources:

"Saudi Arabia: Iraq, Iran, the Regional Power Balance, and the Sectarian Question" by F. Gregory Gause III. Link.

"Iran's Foreign Policy Strategy After Saddam" by Kayhan Barzegar. Link containing some of the article.

"Iran’s Rivalry With Saudi Arabia Between the Gulf Wars" by Henner Fürtig.

"Battle for Bahrain: What One Uprising Meant for the Gulf States and Iran" by Brandon Friedman. Link.