The Tories denounced Powell's speech because it made race into a political issue and disrupted a fragile political consensus that benefited both them and the Labour Party. Britain's economic recovery following World War II depended on the maintenance of the peace between classes that had existed during the war, and on the exploitation of workers from the colonies, who provided a large labor force but did not receive much help from the state in the way of housing and welfare.

There was a bipartisan effort in government to maintain this situation by avoiding emotional politics about race or labor. Although Labour was publicly committed to racial equality, trade unions were not, and making race a big question threatened to open this fracture. The Tories on the other hand were obviously not keen on immigration, but wanted to avoid charges of racism, and like Labour avoided outright engagement on the topic of race through support for slow racial inclusion. Both parties viewed the question of racism as a political liability, and did their best to manage public opinion on the topic rather than following it.

For example, the Conservative's Shadow Home Secretary Quintin Hogg recommended supporting the 1968 Race Relations Bill which outlawed racial discrimination in private employment, loans, housing, insurance. Hogg argued that the Conservative Party should approach the issue of racism "coolly, and cautiously, but liberally. We should not shut the door against extension [of laws against racism]. We should take the credit for the conciliation machinery. We should, on the other hand, sound a warning against excessive precipitancy. We must lead public opinion, not drag behind it, or bulldoze it."

Therefore both parties supported a slow, limited, and unemotional approach to the issue of racism and immigration. Powell's speech was a populist attack on this bipartisan politics. As he said in another talk, there had opened "a deep and dangerous gulf in the nation," between "the overwhelming majority of people throughout the country on the one side, and the other side a tiny minority." Powell saw himself as leading a revolt of the silent majority who were deeply uncertain about Britain's transformation into a liberal, postcolonial state, which was happening under the aegis of both political parties.

Source:

Camilla Schofield, Enoch Powell and the Making of Postcolonial Britain. Cambridge: Cambridge University Press, 2013.