Watched a documentary about this and I believe the Conservative Party at least immediately tried to distance themselves from the comments. I think he was persona non grata in the party until he left.

The purpose of the speech was populism and naturally enough it was popular in certain quarters. There was a spontaneous demonstration in support of Powell by the highly unionized London dockers, for example. If OP grew up in Wolverhampton he or she may well have come across oldtimers who remember it as a "a great bright light." I was a child in London at the time, and I remember the speech being greeted with revulsion. Given its content it deserved to be. Much of it was taken up with unsubstantiated allegations of the harassment of an elderly woman in Powell's constituency by her immigrant neighbours. In no way should the speech be taken as unusually blunt, saying the unsayable, standing against political correctness etc. and all the usual excuses. It was a calculated attempt to stir up racial hatred and so ferment the problems Powell was warning against.

There was a streak of humbug too, in that as Minister of Health Powell had encouraged New Commonwealth immigration to recruit workers for the National Health Service. The speech was a calculated attempt to advance Powell's career by usurping the leadership of the Conservative Party through a populist groundswell. He was quite rightly fired from the Shadow Cabinet at once, and no groundswell emerged.

There were elements of buffoonery to Powell's character which with the distance of time can make him appear Quixotic. He was in fact arrogant, opinionated and ambitious.

Further reading Paul Foot (of whom I am no unqualified admirer) wrote a good study of Powell's career for Penguin books in the 1970s.