There have always been radical individuals. I mean, if we want to call the Kharijites radicals, we can say almost since the beginning (they've recently come more into focus since many have compared them to the Islamic State). If we want to say that the original Arab raiders who conquered Arabia and ultimately all of what's today the Arab and Persian worlds were radicals, then we can say radicals have been part of Islam since the beginning. (Incidentally, we can also talk about radical Christians and radical Protestants, two since the beginning as well--I'm blanking on the name of the Christian group that tried to get themselves martyred, but Romans generally saw Christians as antinomian and, well, a problem).

But I suspect that's not what you're asking. You're asking about modern "jihadi terrorism" and its relatives. This is a much more recent phenomenon. An example that I like to give is that Palestinian "resistance" to Israel was largely secular until the 1980's. The Mufti of Jerusalem in the 1920's and 1930's was of course instrumental in the riots and pogroms of the era, but that genealogy isn't very strongly connected to modern Islamic terrorism. It was more a nationalist movement led by a religious leader, than a religious revivalist movement with violent elements. By the 50's, 60's, and 70's, the Palestinian leadership infrastructure (Fatah, PFLP, DFLP) was clearly nationalist and vaguely secular socialists. They committed acts of terrorism, certainly (the infamous killing of Israeli athletes at the 1972 Olympics was only one example of many), but it would be a mistake to see them as groups that saw themselves as inspired by Islam, groups that encourage piety (never mind demanded it), or groups that gave religious justifications for their violence. In Palestine, those groups only emerged in the 1980's: Islamic Jihad in 1981, Hamas in 1987. So in Israel, clearly, the answer is the 1980's.

In Lebanon, Hizbollah only emerged in the 1980's, and only in reaction to the Israeli occupation of South Lebanon (though it gained power in part because it was support by Iran). Iran was a secular state and, if you study the protests that ended up overthrowing the Shah, it's unclear that an Islamic movement is going to take over until very late in the Revolution. Before that, the pious protestors were just one group of many in the protests. Most modern Shi'a "radical" groups can be tied the Iranian regime's post-1979 support in some way (the Houthis, who are Zaydi rather Twelver, are an obvious exception--they didn't start receiving serious Iranian support until more recently).

Most violent modern Sunni groups that you may have heard of and "threaten Western interests" have a slightly older genealogy, but one only dating back to the mid-20th century. The key group is the Muslim Brotherhood in Egypt, and the key figure is Sayyid Qutb, who it should be pointed out was never even the Brotherhood's leader (some people call modern Jihadism "Qutbism"). But the key thing to remember is that Qutb is a mid-20th century figure. He urged a violent overthrow of the secular Egyptian state (and similar states led by "hypocrites", a key term for Qutb), and was ultimately caught and killed by the Egyptian State in 1966. The Muslim Brotherhood was and is one of the most important civil society groups in the Arab World (Hamas is a descended of the Muslim Brotherhood as well).

For more on Qutb, Osama bin Laden, and the relationship between them and Muhammad ibn Abd al-Wahhab (perhaps you've heard of "Wahhabis"), see my earlier explanation "What is the history of (modern) Islamic Extremism?", particularly paragraph five (about Qutb) and paragraph six (a thin paragraph how Osama bin Laden helped bring Qutbism and Salafism together, but most people who support Al Qaeda don't actually quite believe in Salafism--for more on this, I strongly recommend Charles Kurzman's The Missing Martyrs).

But while we saw radical Islamists, who wanted to instate shari'ah law (the Ottomans ruled by a mix of religious shari'ah and secular kanun for centuries), except in a few places like Egypt, they were generally marginal groups. In Turkey, there was the tiny Tijani group defacing statues of Ataturk as idolatry (they were the first modern statuary in the country), but it never really went beyond that.

Thomas Hegghammer shows that modern, trans-nationalism jihadism didn't really get its start until the 1980's War in Afghanistan. Before this, what radicals there were were generally only interested in local fights. This was fairly small, but it was still a couple of thousand fighters. This is really where "modern jihadism" comes together. It's no coincidence that Osama bin Laden and Al Qaeda really got their stats there (they later combined with some older, originally more domestically inclined groups already existent in the Arab world--most of these old Qutbist factions). Now, not all parts of Islam you'd call "radical" fit under the label of modern jihadism. The Taliban, for instance, who were in post-war Afghanistan and allies of Osama bin Laden don't come from the same mold. For more on this, see Charles Kurzman's very short and very readable article, "Bin Laden and Other Thoroughly Modern Muslims" (in short, Kurzman argues that Bin Laden and his ilk are a modernist while the Taliban are traditionalists).

So, in sum, the second half of the 20th century is probably the best bet. In Egypt and a few other places, you start to see Qutbists emerge and become a serious problem in the 50's, but in most places you don't see radical Islam "becoming a serious problem" until the last quarter of the 20th century. You have the Islamic Revolution in Iran and the siege of Haram al-Sherif (and the subsequent tightening of Saudi Society) starting in 1979; you see the emergence of a variety of violent Islamist groups in the Levant in the 80's; you see the first big Islamist attack on the U.S. in 1983 (this was by a Shi'a group) followed by more sustained attacks on the U.S. interests in the 1990's by al Qaeda and al Qaeda-type groups (1993 the first World Trade Center bombing, the 1998 embassy bombings, U.S.S. Cole bombing in 2000); you have important ideologies (Qutbism+Salafism+importance of attacking outside groups, rather than domestic "hypocrites") coming together in Afghanistan, as well as important networks (preachers+fighters+fund raisers) in the 1980's as well.

In short, as Charles Kurzman says, the kind of radical Islam I believe you're asking about is a thoroughly modern problem.

Warning: long answer!

So before I get into this, because I do have some historical examples that you might find interesting, I do feel compelled to at least touch on the definitional issues involved in your question particularly around the words "radical," "Islam," and "problem." The issue isn't so much that these words are subjective, because historians make relative subjective judgements all the time, it's that they have very different meanings based on time and place and that these meanings have only a limited relationship to one another.

To avoid spoiling some of the examples I'll discuss below imagine if we were to ask "at what point in history did radical Christianity become a serious problem?" If you're the Romans in 50 AD you can plausibly say "from the start!" but the challenge of a radical fanatic (from the Roman perspective) like Jesus Christ is different and poses different kinds of problems than the radical Christianity of the Crusaders. That in turn poses a different problem than John Calvin. They also have very little to do with one another! So even though different peoples at different times might have perceived a problem with some particular group of radical Christians these groups may not have had any ideological kinship whatsoever. Calvin may have embraced the early Church, in rejection of the Crusaders, but Christ may not have recognized the violence of the Calvinists. That sort of thing.

There's at least two solutions to this problem. One is to get way more specific with the kind of phrasing you use. So you're question might instead read something "What are the intellectual and historical origins of the late 20th century violent Sunni Jihadist movement, particularly Al-Qaeda?"

I'll touch on that one specifically, but I don't think that's quite as fun as just discussing some of the more interesting "radical" groups throughout Islamic history, just as I'm sure some people in this sub might have fun talking Anabaptists or the Taiping as interesting examples in Christian history.

Feel free to ask about any particular example that strikes your fancy as being interesting:

So let's begin!

Islam under Muhammad

Essentially from the start Islam was a challenging religion to the people of Arabia that posed a serious problem to the established order. Even though Muhammad was a member of the most powerful tribe in Mecca, Quraysh, it was made clear to him that his preaching was not welcome and so he was more or less forced to flee to Medina, helped along by an invitation from that city for Muhammad to mediate their own political disputes and, as a result, to effectively become the leader of their community.

While Islam has connections with a number of religious traditions and customs that would have been familiar in Arabia-- certain aspects of pagan ritual, Jewish dietary law, Christian figures, etc. -- it nonetheless demanded quite radical changes in society and was, more or less from the start, involved in violence with neighboring communities for one reason or another.

The Conquests

After Muhammad's death was when the true expansion of his state began. By the middle of the 7th century the Arabs had smashed two of the largest and oldest empires on the planet.

This is actually the one example that I might push back against. Gibbon, writing in the 18th century, felt comfortable describing these conquests as Islamic and headed by religious fanatics who basically out of sheer religious fury were able to conquer these lands. The past century or so of scholarship has really cast doubt on that idea.

The Arabs were unified by Islam but were not uniformly Islamic, they don't seem to have been viewed as being particularly outrageous in their religion, and the conquests, considering their scale, weren't particularly brutal. Some have gone so far as to say that certain disaffected Christian groups may have actually welcomed their Islamic conquerors as a more tolerant alternative to the Orthodox Roman Church. I think that's not at all supported in the contemporary evidence but the point stands that this was not viewed as being religiously all that unusual.

The Kharijites

The fourth succession to Muhammad has caused arguably some of the greatest debates in the religion. Muhammad's nephew and son and in law Ali, according to his supporters, was supposed to have been the first and primary successor to Muhammad but had to settle for being the fourth of the rightly guided caliphs. A cousin of his predecessor Uthman disputed this and the result was the First Fitna, a sort of Islamic civil war.

When the two sides met in battle Ali decided that he actually wanted to settle the dispute by arbitration, but many of his supporters rejected this as blasphemy or heresy. Ali's refusal to determine the divine will by combat, thus proving that he was the chosen successor of God's messenger, was beyond the pale and as a result they "went out" from his camp, or in Arabic verb "kharaja" to go out, thus Kharajites.

Ali defeated most of this group at a later battle, but he was eventually murdered by a Kharajite. Ever since the title has been applied to a number of groups, usually by their detractors, which have only a tenuous relationship to one another but do, in that description, often have several things in common.

One notable commonality between historical Kharajite groups has been their willingness to declare what they few as not being valid Islam and a willingness to kill accordingly. The Azraqites are one notable example of this who went so far as to say that failing to actively join them in their beliefs was itself a mark of unbelief and rendered all others lawful to kill.

As W. Montgomery Watt writes: "From the first the militant Kharijites had engaged in the practice of isti'rad [NB: lit. exclusion], that is, the killing of those, even Muslims, who did not accept all their views. Their justification for this was that such persons, although professing Islam, were not really Muslims....The Azraqites...went a step further. To be a true Muslim in their eyes it was necessary not merely to hold the beliefs they held but also to go out [NB: there's "kharaja" again] to their camp in the mountains. To 'sit still' at home was a mark of unbelief. Moreover, since the 'unbelievers' thus defined were the enemies of the 'Muslims', the Azraqites believed it lawful to kill them and enslave their children."

I think you can see from the above why some contemporary observers of radical Islamic movements in our own time have drawn connections and used some of the same terminology to describe modern groups. Nonetheless, the only practicing group of Muslims that actually has any direct connection with the historical Kharijites are the Ibadi muslims of modern Oman. In our own times Ibadi Muslims are a sect that is regarded as having some of the least affinity for radical extremist violence.

This kind of historical irony also applies to the latter day existence of my next example the...

Assassins

Let me preface this by saying that the Assassins are kind of a specialist topic insofar as they draw a huge amount of attention as you might imagine, so there's a huge amount written about them of which I have read very, very, very little. I am not an Assassins expert, just as I am not a Crusades expert.

That being said-- the Assassins were a militant order of a particular branch of Shia Islam (i.e. that traced the descent of its leadership back to Ali) that actually at one time in predecessor sect ruled Egypt as the Fatimid Caliphs. The Assassins were ruled by the descendants of Nizar who was basically deprived of the throne and then fled Egypt.

To cut a long story short, the Assassins, who had an interesting power structure, basically set themselves up in independent castles across Syria and parts of Persia and, most famously, seem to have conducted political assassinations.

The most famous such assassination was the killing of Nizam al-Mulk, longtime vizier of the Sunni Seljuk Turks and therefore probably one of the most powerful men in the world at the time. As an aside to your question on the Islamic Golden Age, Nizam al-Mulk was also critical in that Golden Age as he tried to promote a kind of religious tolerance as well as making huge educational reforms that massively expanded the number of very-nearly-university-esque educational institutions in Islamdom, the eponymous "Nizamiyya".

The Assassins were nearly wiped out by the Mongols, but as I mentioned their latter day history is perhaps a bit unexpected because the ultimate lineal descendants of the Nizari Shia "Assassins" are the modern followers of the Aga Khan, who basically spends his days promoting peace and education and whose sect of some 15-25 million people promotes reason, pluralism and social justice as cornerstones of their faith and have no contemporary or recent historical violent movements that I'm aware of.

edit: to be continued!

Forgive me if I'm mistaken, but as I understand it there is no scholarly consensus on the matter and there is a lot of discussion on the how and when of radical Islam. I don't think there was any one point where it became a problem, more like a series of events that took place that set up the situation we have today. Also, it is somewhat difficult to define what constitutes a "serious problem". After all there's a pretty big gap in time between the Islamic Golden Age and the modern version of radical Islam. I'll focus in on the current situation and attribute modern levels of violence, destruction, and deaths as what was meant by the term "serious problem".

One example is The Rise of Islamic Fundamentalism which argues in a broad general sense that the late 1960s point in time saw a shift from Arab nationalist/populist regimes and forces to Islamism. There are so many geo-socio-political causes for this shift, the oil boom of the 70s, the growing threat of Communism, and the rather arbitrary borders drawn after other conflicts last century, and especially the '67 conflict with Israel, all of which led to Islamism being seen as "the answer" to those problems.

Historically this area has always been in contention as it is important for trade and for buffer zones, and the modern era has seen the region used as a sort of theater for superpower proxy wars. There's just so many interests pulling in so many different directions that it's hard to pinpoint any one cause for conflict. It's no wonder that natives to the area are latching onto Islamism as a life raft as it's arguably seen as the only non-outside influence.

The problem with modern radical Islam is that it didn't appear out of nowhere. I would argue that radical Islam began in the early 20th century with persons like Abu A'la Maududi, Hasan al-Banna (the founder of the Muslim Brotherhood) and then, a little bit later on Sayyid Qtub.

I want recommend you Emmanuel Sivan's "Radical Islam: Medieval Theology and Modern Politics". It's a really good and well-written book tracing the origins of "modern" radical Islam. I put modern in quotation marks since the book was published in 1985 and therefore cannot adress anything that happened past that year. Sivan argues that radical Islam did emerge as a reaction to modernity and that radical Islam basically is fighting against modernity.

Sivan's main argument for the emergence of radical Islam is that after the Arabic/Islamic world became independent from Western colonialism modernity did not deliver in the way it did in the Western world. Expectations of catching up economically and in terms of standard of living were not met and at the same time, as a result of mass media, popular culture, and education Islam became virtually absent in society. As a result disappointed young men turned towards radical thinkers like Maududi or al-Banna.

If we look at the main concern of the early thinkers the main concern, which is nationalism, becomes clear. Maududi coined the term "Modern Jahiliyyah". Jahiliyyah means "pre-Islamic ignorance" and describes the state of being before the emergence of Islam in the 7th century. He argues that the modern Islamic world has reverted into Jahiliyyah because the Shari'a as the godly order was replaced by constitutions and laws, inspired by the West.

Maududi's thoughts about modern Jahiliyyah are picked up by Sayyid Qtub in the early 60s in his book Signposts along the road (Ma'alim fi al-Tariq). Qtub argues that the Sharia has to be brought back as the only kind of order. He further argues that the fight against Arab nations would be Jihad. In theory Islam forbids inter-Islamic warfare. But Qtub brings in Ibn Taymiyyah, a medieval Islamic theologist and scholar who issued several fatwas against the invading Mongols around 1400. Qtub, and later Muhammad abd-al-Salam Faraj (he was part of the group that assassinated Anwar al-Sadat in 1981 and wrote their ideological base "The neglected duty"), argue that the modern rulers of the Islamic world are no Muslims but apostates because they do not rule according to the Sharia but rather according to manmade laws. Since they are apostates they are no longer Muslims and therefore it is the duty of every Muslim to fight and wage Jihad against their unjust rule (the neglected duty Faraj is talking of is Jihad). They rely heavily on Ibn Taymiyyah's so-called Mardin fatwa which, as far as they understood it, justified fighting against the invading Mongols even though they had converted to Islam. What they did not know and only came to light in 2010 is that at some point an error occured while copying the Mardin fatwa. The orignial version was forgotten and the version with the error was distributed.

The version that was used by Qtub and Faraj ends like this: “…while the non-Muslims living there outside of the authority of Islamic Law should be fought as is their due.”

The correct version is as follows: “The Muslims living therein should be treated according to their rights as Muslims, while the non-Muslims living there outside of the authority of Islamic Law should be treated according to their rights.”

There was a conference in 2010 where the Mardin fatwa was discussed and this came to light/was made public. The following link leads to a short paper in which this is discussed and you can find the full text of the fatwa in English translation: http://muslimmatters.org/2010/06/29/the-mardin-conference-%E2%80%93-a-detailed-account/

Coming back to Qtub. His usage of Ibn Taymiyyah, argument that the modern rulers should be treated as non-Muslims and his justification became one of the most important pillars of modern radical Islam and hugely influenced groups like Al Qaeda.

Also very important to note is that almost all of the ideologists of modern radical Islam were self-taught and did not have a formal religious education. Maududi started as a journalist, Qtub was a teacher and author, Faraj was an engineer. Hasan al-Banna was a teacher as well as an Imam and was learned in his local mosque.

Coming to some reading recommendations: As already mentioned:
Sivan Emmanuel, Radical Islam: Medieval Theology and Modern Politics

Also very interesting: Keppel, Gilles, The Roots of Radical Islam

On the topic of Ibn Taymiyyah: Yahya Michot, Muslims Under non-Muslim Rule

I would also recommend having a look at Qtub's Signposts as well as Faraj's The neglected Duty. Both are pretty short and give a great insight in the thinking of these radicals. Even without being knowledable in Islam one can recognize the flaws in their thinking and sense their hate and resentment. You can find them without problems via Google in English translation.

I hope this post answers at least some part of your question.