Briefly, revolutionary nationalism in Third World countries is a catch-all term for a lot of anti-colonial sentiment that generally borrowed socialist rhetoric in order to appeal to international anti-colonial actors that might give them aid, usually the USSR or Chine. The Khmer Rouge was one such movement, and a lot of its socialist rhetoric can be viewed in this context. What I mean to say, if we were ever to describe a regime as Nationalist and Socialist, the Khmer Rouge fits it perfectly.

They were intensely, radically, and unapologetically nationalist - and this is the root of why they were so violent. The ideology of Angkor was unequivocally palingenetic, obsessed with the rebirth of a nation embarrassed by both its neighbors through loss of territory and by colonial administrators. They hated foreign-ness, more than anything. This characterized the early days of the regime, and explains some idiosyncracies of their movement that you wouldn't consider typically communist - such as their alliance with and support of with the Khmer monarchy that had been deposed by the US-backed Lon Nol.

Consider this speech by Pol Pot the day they took over Phnom Penh:

The brother and sister combatants of the revolutionary army … sons and daughters of our workers and peasants…were taken aback by the overwhelming, unspeakable sight of long-haired men and youngsters wearing bizarre clothing making themselves indistinguishable from the fair sex …. Our traditional mentality, mores, traditions and literature and arts and culture and tradition were totally destroyed by U.S. imperialism and its stooges…. Our people’s traditionally, clean, sound characteristics and essence were completely absent and abandoned, replaced by imperialistic, pornographic, shameless, perverted and fanatic traits.

There were three main foreign enemies that Pol Pot wanted to eradicate from the Khmer mentality - the Chinese, the Vietnamese, and the Westerners. The first two were physically present on Khmer soil, and their removal was violent. Pol Pot himself gave a speech in 1978 praising the fact that 'not one seed of Vietnamese remained in Kampuchea'. Looking at witness statements, we see how that anti-Vietnamese sentiment was expressed:

As you all know, during the Lon Nol regime the Chinese were parasites on our nation …. The population of each village will be divided into a Chinese, a Vietnamese and a Cambodian section. Some of you are not Cambodian, stand up and leave the group. Remember that Chinese and Vietnamese look completely different from Cambodians. About ten people stood up and walked to the place reserved for them; ‘‘Are there any more?” No one stood … then the four guards were told to go through the crowd. Anyone whose face looked foreign was different was dragged out.

A good source for these witness statements and speeches is At the Edge of the Forest: Essays on Cambodia, History, and Narrative in Honor of David Chandler

So the early violence was directed at these two enemies - the Chinese and Vietnamese. However, the Khmer Rouge also had two more beliefs that gave it its violent character.

The first was the belief that Cambodia, in order to be great again, needed to be cleansed of foreign, especially Western, thought and beliefs. The Angkor held that those who lived in cities were corrupted by the Western 'bourgeoisie mentality' and targeted intellectuals, business owners, or those who had too much to do with the former colonial administration or foreigners.

There was also a belief that after a certain age, and in certain groups of people, re-education simply wasn't possible anymore. This is part of the reason why the Khmer Rouge relied so much on youth cadres (their minds were still pure). Where reeducation wasn't possible, killing was seen as a way to save resources and keep groups, especially urban groups, in line. The Khmer Rouge called this group 'New People', and when they weren't outright killed the reeducation process was so brutal (they weren't allowed to own land, were given very sparse rations, and forced to labor ten-twelve hours per day) that they often died of disease or starvation anyway. These people are what the rather infamous KR motto "To keep you is no benefit. To destroy you is no loss." are referring to.

The second was a concept of Year 0, referring to the year before the French Revolution before the French colonized the region. Cambodia at that time was agrarian, so the KR thought that the socialist state should also be agrarian. This required deindustrialization, and the evacuation of cities. Many people died in this process as well, for similar reasons as above. The deindustrialization effort wreaked havoc on the Khmer economy, and Pol Pot's incredible paranoia about a Vietnamese invasion made resources even more scarce as more and more was devoted to the military. The lack of resources led to more death, and eventually, more peasant displeasure at the regime - of course, when you've already so wholly adopted violence as a means to an end, what's the easiest solution to the peasants starting to resist your rule? More violence against these new 'anti-revolutionaries'.

If you want to get more into the racial theories of Kampuchea and Angkor, I recommend:

The Pol Pot Regime: Race, Power, and Genocide in Cambodia under the Khmer Rouge, 1975-79

I agree with much of /u/m4nu's answer, and I should say that I had looked at this question for a few hours and found myself without enough time to answer what seems like an easy question but has many possible and relatively complex answers...

Comparing instances of mass death, whatever the ideology which may have underpinned them, is always a tricky business. If we look at the Khmer Rouge as a movement comparable with other socialist or communist governments, then we are comparing apples with apples but ... perhaps different kinds of apples. A Granny Smith versus a Pink Lady?

The Stalinist Purges, or the horrors of Mao’s Great Leap Forward… these policies resulted in far more deaths than Cambodia’s entire population in the 1970’s.

But, as the other answer to this question has suggested, there were peculiarities to the Khmer communist movement that resulted in certain catastrophic results, perhaps, as you suggest, with an effect ‘worse’ than either of these movements or the human cost of revolution in Vietnam as well. So why might this be the case?

Well lets briefly touch on the death toll. While there is some debate around the numbers, according to the most well put together studies and estimates, and studies of those studies and estimates, its fairly safe to say that the death toll during the regime was somewhere close to 2 million. Up to half of that number may have been the result of direct violence, or government sponsored mass killings. The other half being the result of disease, starvation and over-work. This, as you’ve mentioned in the question, equates to about one fifth or one fourth of the entire population depending on the estimates of how many people were living in Cambodia in 1975. That is the statistic that many people would highlight if they were trying to say that the Khmer Rouge were the ‘worst of the worst’. While Stalin and Mao killed millions more, they didn’t decimate their entire population to quite the same degree.

Well, why did they do that?

As the other answer outlined, there were certain ethnic groups which the Khmer Rouge had a particular interest in eliminating. The Vietnamese were certainly one of these, as were other minorities such as the Muslim Chams, however there is some academic debate about whether this was a result of an ideology which stressed the ethnic or racial differences between the Khmer and these other minorities, or rather just the way that the Khmer Rouge interpreted Marxist-Leninist-Maoist ideology in a way that precluded minority ethnic groups from existing within their idealised revolutionary society. See Steve Heder’s review of Ben Kiernan’s Pol Pot Regime, for a rundown of this debate.

But the awful suffering of these minorities, regardless of the ideological framework supporting it, only represents a fraction of the total death toll that we outlined before. Perhaps one hundred thousand out of two million, or roughly 5 percent.

So I think this is not quite where our idea about why the Khmer Rouge were so terrible should rest.

I would contend that the primary reason that the Khmer Rouge were compelled to commit the vast majority of their crimes against humanity, including genocide against minority groups, but the bulk of these crimes being committed by the ethnic majority against the Khmer themselves, well the main reason behind this was simply the scope and speed in which the leaders of the Khmer Rouge intended their revolution to be carried out. Historian Henri Locard has suggested that the Khmer Rouge felt a need to make their revolution as radical as possible in order to go above and beyond the other socialist revolutions that had occurred in the 20th century, they borrowed other revolutionary language but doubled down, repeatedly calling for the need for a super great leap forward, for complete ‘independence mastery’, to commit so totally to vague ideas of complete collectivisation that they emptied all of the cities in Cambodia and abolished money… something that Lenin had thought of but had been convinced was an impactable idea.

To stand an entire society on its head, essentially overnight, as well as committing to destroying all enemies and potential enemies of the regime as a matter of policy… smashing them. This is why the Khmer Rouge were so oppressive and violent. Because they were committed to a complete societal revolution, one based on Marxist-leninist-Maoist ideology, but also with uniquely Khmer characteristics, such as the frame of Theravada Buddhism. The CPK viewed ‘counter-revolutionaries’ a framing of enemies shared across many revolutionary societies, within a binary of who could and who could not attain a revolutionary state of mind, a ‘proletarian consciousness’; satiaramma. Those unable to do so were considered a enemies of the regime, and disposed of through various hierarchies of the regime.

The fact that the Khmer Rouge had enslaved the entire population of Cambodia, had exposed each and every citizen of their new society to the kind of potential elimination as they did to the members of their own party… this resulted in an entire culture of potential death for the slightest infraction against the state. With lower ranking village chief and cadre given the impetus to try and purify the revolution, many uneducated and highly indoctrinated men and women considered murdering these enemies of the revolution as a duty that they could not ignore, or perhaps face a similar fate.

For an anthropological explanation of the question you have asked, as opposed to a political or historical one, I would suggest Alex Hinton’s Why Did They Kill?

I’m unable to go too deep into a kind of comparative answer regarding the other socialist governments you mention because I can’t say they are within my area of expertise, but as I said earlier sometimes these comparisons are not very helpful, the raw death tolls and statistics are one thing but trying to measure the suffering between groups isn’t quantifiable. “Why were they worse” is a very difficult question to answer.