Wouldn't the lower classes of Chinese society dwarf those in power? From what I understand, the Red Guard was feared, but with millions struggling with famine and disillusionment with a regime, wouldn't many of those troops joined some kind of counter revolution against him?
Try asking in /r/communism ^/s
Serious answer: Your question makes a couple of presuppositions which I think are inaccurate, including the implication that "the people" regarded themselves as distinct, and in opposition, to "the regime," which they blamed for their problems.
While it's true that many people suffered, it was unusual to blame higher levels of government or the Communist system itself. The government had enormous prestige through the 1950s, most of it well-deserved. Opposition voices had also been sharply rebuked by the 1956 Anti-Rightist campaign(as well as many of the economists and statisticians who would have been able to point out how terrible an idea the GLF was).
Likewise, resistance was limited to relatively local acts of disobedience, like soldiers opening grain stores during the famine. Jung Chang's memoir recounts several instances of people going to the capital to appeal against local officials, for example.
The only really large-scale resistance I know of happened in minority areas, like Tibet(which is relatively well-known.) In Xinjiang(the Muslim part of China) there was a mass exodus during the cultural revolution, when thousands of Soviet dual citizens rushed for the border.
Sauce: Spence's History of Modern China, Jung Chang's Wild Swans, and several years of living in China.