I'm going to focus here on late 1800s-1970s; other views go back to the Medieval era.

Criticism of Islam has a long history in West Europe. Most of the earlier criticisms come from figures in politics and academia. They are not that different from what many say today. Take this from scholar and colonial official William Muir's speech at Cambridge University in 1881:

Some, indeed, dream of an Islam in the future, rationalised and regenerate. All this has been tried already, and has miserably failed. The Koran has so encrusted the religion in a hard unyielding casement of ordinances and social laws, that if the shell be broken the life is gone. A rationalistic Islam would be Islam no longer. The contrast between our own faith and Islam is most remarkable. There are in our Scriptures living germs of truth, which accord with civil and religious liberty, and will expand with advancing civilisation. In Islam it is just the reverse. The Koran has no such teaching as with us has abolished polygamy, slavery, and arbitrary divorce, and has elevated woman to her proper place. As a Reformer, Mahomet did advance his people to a certain point, but as a Prophet he left them fixed immovably at that point for all time to come.

French historian Andre Servier in the 1924 work L’islam et la psychologie du musulman:

Islam is by no means a negligible element in the destiny of humanity. The mass of three hundred million believers is growing daily, because in most Musulman countries the birth-rate exceeds the death-rate, and also because the religious propaganda is constantly gaining new adherents among tribes still in a state of barbarism.

Such opinions of disgust and fear could be found among many famous people of the period, Winston Churchill among them. Considering that during this time Islam was dominant in a lot of regions where conditions were backwards compared to Western Europe (which was comparatively well off and in charge of several colonial empires), this is not surprising. These types of views were by no means limited to Muslims: non-Muslims in Africa and Asia were commonly seen as inferior too.

Yet many figures also showed a deep interest in Islam that was not just contemptuous. Historian William Watt published analyses about Islam's founding in the 1950s that are still acclaimed by both Muslims/non-Muslims. See Muhammad at Mecca and Muhammad at Medina. Other European works relating to the Middle East and South Asia were regularly published, and this reflected European interest. However, many saw them as biased, patronising, and Eurocentric (Edward Said outlined this position in his famous 1978 book Orientalism)

Regular Europeans held feelings of hostility towards Muslim immigrants before the 1980s. Clashes between major far-right groups and Bangladeshis/Pakistanis were not uncommon in Britain. However, it seems like most of this tension was rooted in race. Non-Muslim black immigrants to Britain were also targeted by racists, most famously in the 1958 Notting Hill Riots