Because of the 20 year rule I’m going to limit my information to pre-1995. This mostly answers your question about the origin of Al-Qaeda, but it doesn’t answer the implied question of why Al-Qaeda started to fight the US and the West. That evolution started to happen right around and mostly after 1995.

Okay, Al-Qaeda has its roots in Peshawar, Pakistan in the 1980s. In 1979 the Soviet Union launched an invasion of Afghanistan in order to prop up a pro-Soviet government. The fight against the Soviets turned into a sensation throughout the Arab world (Note, Afghans are not Arabs). While Afghans fought the Soviets, thousands of Arabs volunteered to go join the fight in Afghanistan. The trail from the Persian Gulf and Saudi Arabia to the battlefield went through Pakistan, and the last large city before the fighters crossed over to Afghanistan is Peshawar. This is the city in which Osama Bin Laden took up residence throughout most of the 1980s. Before 1991 (and I guess especially after), the Bin Ladens were the 2nd most prominent family in Saudi Arabia after the House of Saud. Mohammed Bin Laden, Osama’s father, was the founder of an extremely large construction firm that won many prominent contracts as the Kingdom built itself up in the 1950s, 60s, and 70s. Osama took to the cause of Jihad and took off for Peshwar. He operated an organization called the “Services Office”, which organized and financed Arab fighters in Afghanistan. Make no mistake, this was not a simple hotel that hosted Arab fighters entering and exiting the battlefield. This organization provided weapons, training, and funding for warriors. Bin Laden even led a small guerilla band late in the conflict. In 1988 took his experience, resources, and small cadre of warriors and started a group called Al Qaeda, although this was completely unremarkable at the time.

It was during this period that Bin Laden was radicalized. He was always a pious man, but sometime during the 1980s he crossed that line from piety to radical. After the Soviets pulled out of Afghanistan in the late 1980s, Bin Laden found himself to be arguably the most successful and famous Arab mujahid in the world. His co-operator, and later rival, of the Services Office, Abdullah Yusuf Azzam, was killed in a car bombing in Peshawar in 1989. This brings us to one of the great crossroads of Bin Laden’s life. In 1990 Osama was a fighter without a cause. He had relocated back to Saudi Arabia after the end of the Soviet war. Then in August of that year Saddam Hussein had invaded Kuwait. Possessing a hatred of Hussein, in part for being a secular socialist leader of an Arab nation, Osama offered the King of Saudi Arabia use of his mujahideen network to fight back the Iraqis, protect the holy cities of Mecca and Medina, and retake Kuwait. This offer was curtly refused, and 500,000 American soldiers soon entered the country. This enraged Bin Laden. He believed defense of Mecca and Medina should be left to his holy warriors, and that the Saudi king was merely a puppet of the American government. Speaking his mind to this effect, Bin Laden was banned from Saudi Arabia and relocated to the Sudan.

This is where we really skirt the 20 year rule. With what has been learned on the battlefield since 2001, it’s clear now that by 1995 Bin Laden had been radicalized, had built up an advanced terrorist network, and had decided that the House of Saud was a corrupt leadership polluting the holy land. However, the real power behind the Saudi throne was the United States and that the House of Saud could never be overthrown without removing the US from the Middle East. The planning for later terrorist attacks against America had already begun by this point.

Source: Ghost Wars and the Bin Ladens by Steve Coll. Also, the Longest War by Peter Bergen