The Iranian Hostage Crisis was probably the biggest factor. This was from 1979-1981. 9/11 just refreshed what was already there. Orientalism probably had a big part in this. There existed an idea that Muslims were very different from those of us in the Christian West and so it was pretty easy to become attached to the idea that it was Islam that was responsible for the Iranian Hostage Crisis rather than any political issues. Those feelings were ignited again with 9/11. This is explained in greater detail in Edward Said's books.