The Khmer Rouge's goal was not so much to establish a blank slate, but to return Cambodia to a "mystic state" and recreate what Pol Pot viewed as the essence of pre-Western contact Cambodia. This came largely as a reaction to the US invasion of Cambodia and the huge loss of life that it caused, as well as a general sense of anti-colonialism and anti-Western sentiment. I'd keep in mind as well that Pol Pot's image of a "mystic past" did appeal to a lot of Cambodians. It was an image of a country not torn by war, not invaded and exploited, and one where people could be happy and hopeful, things which hadn't been the case in Cambodia for decades.

Inherent in this, though, is also a large degree of Cambodian nationalism, that sense of what Cambodia was and could be. Pol Pot, as I said, presented an image that appealed to people because it resonated with their own sense of what it was to be Cambodian and what "Cambodia" meant. Coupled with this, though, was the material and cultural idea of "Cambodia," of which something like Angkor Wat was a part. It belonged in that "mystic past" where Cambodia existed as a powerful state independent from foreign - and particularly Western - influence.

Basically, then, the reason the Khmer Rouge didn't destroy Angkor Wat and other such structures was because their goal was never to remake Cambodia as such. What they wanted was a return to an archaic Cambodia, one which they saw as following Maoist ideals of equal participation in agrarianism, but one which was undoubtedly Cambodian and which preserved the Cambodian identity.

I hope that helps, and if you have any more questions, feel free to ask!

The Democratic Kampuchea flag had a three-towered representation of Angkor Wat on it. Pol Pot was quoted saying "If our people can build Angkor, they can do anything". So obviously they were quite proud of this heritage and intent on keeping it.

Here's a post I made about them a good while back that might provide some answers.

Pol Pot's revolution was an entirely peasant based uprising. Democratic Kampuchea was an attempt to turn the entirety of Cambodia into a peasant based agrarian society. The Cambodian peasantry didn't know private property. All land and property belonged to the French, the Aristocracy, and the Japanese. The Cambodian peasantry didn't know education. The French purposefully kept literacy and education rates as low as they could to prevent political unrest. Once the Japanese were run out of Cambodia, a new Aristocracy took control and through Prince Shinouk's attempts at neutrality Cambodia began to be infiltrated with Vietnamese. Cambodian peasant relations with the Vietnamese had always been sour as the Vietnamese were better educated and enjoyed higher privileges in Cambodian society than even Cambodian peasants. Eventually a military dictatorship came to power, one that allied with the United States and was complacent in the death of hundreds of thousands of Cambodian innocents. The peasants who had never earned much money were packed into cities to escape bombing runs, left with no money to buy food or provisions, left to starve.

According to the Khmer itself, the US bombing raid was the final rallying cry for the Cambodian peasantry, which fully allied itself with the Angkar, the Khmer Rouge before they fully adopted Marxist/Communist posturing. The Khmer state reflected what the Cambodian peasantry knew: It was an agricultural state, one that favored agricultural labor over education, one that favored full communalization of property, one that favored Cambodian ethnicity, one that favored traditional Cambodian peasant life. Property, education, money, multi culturalism were all symptoms of things that had kept the Cambodian peasants poor and destitute, and in the new Cambodian state, those traits would be eradicated.

Pol Pot was overwhelmingly popular with the Cambodian peasantry. Enough to empty entire cities through force. You know what happened to the Cambodian people afterwards and Pol Pot eventually began to alienate the peasantry through over work and dehumanization and using starvation as a weapon and means of control ("If someone is very hungry, the Angkar will take him where he will be stuffed with food."). When the Khmer Rouge were removed from power through collaborators with the Vietnamese a sizable portion of the Khmer Rouge forces went with Pol Pot into exile. These forces were the ones that came from the provinces within Cambodia that saw the least amount of strife, as certain districts did not suffer nearly as immensely as others. They were also propped up with US and UK funding but their entire goal was to remove Vietnam from Cambodian soil. In interviews post downfall Pol Pot was steadfast in believing that what he was doing was right, and he still had support from a faction of the peasantry. While I've never heard that particular saying in your question, Pol Pot's isolation from the absolute worst situations in the Khmer Rouge, his belief that all that oppose the Angkar were not true Cambodian peasants, and his continued support in exile gave him all the vindication that he needed. Sources:

The Khmer Rouge: Ideology, Militarism, and the Revolution That Consumed a Generation Pol Pot's Little Red Book Brother Number One: A Political Biography Of Pol Pot Pol Pot Plans the Future Nate Thayer's AMA and the reference interview as well.