The murder of 1.5-3 million people under the Khmer Rouge is mind-boggling to me. Given that every single person involved with the basic levels of the regime would themselves very likely have family members at risk or already dead, how did the bureaucracy, military, and economy even function? Was it simply forced, or who did they recruit?
A grim subject, this. I don't particularly enjoy answering these questions about the darker parts of history, but I think it's important for any student of history to embrace the saying "those who do not learn history are doomed to repeat it"... and I sorely hope this is never ever repeated.
Let me begin by trying to answer the first part of your question first... "How did the Khmer Rouge manage to kill a third of Cambodia's population?"
First - segregation and dehumanisation of the "others" as opposed to "us". Or, racial discrimination. It is a powerful, evil, and yet extremely effective tool used by tyrannical regimes. Just like Nazi Germany dehumanised the Jews, the Khmer Rouge did the same thing to the Cham Muslims, who were victims of a brutal genocide that nearly wiped them out. Also like the Nazis, the Khmer Rouge emphasised the "purity" of the Cambodian race and to that end targeted those of Chinese and Vietnamese heritage for ethnic cleansing.
Adding on to the racial discrimination, there was also religious discrimination. But not one religion versus another, no - Pol Pot was an ardent atheist. All religion was banned, and regarding Buddhism alone thousands of monks were killed.
Second - mass deaths due to nationwide starvation, malnutrition, and illness.
Let's start with the most significant part - starvation. The Khmer Rouge envisioned a classless, cashless agrarian state, supported by small industry. To get rid of class, they removed all private enterprise - everything was owned by the state and collectives. The Chinese - who were stereotyped as businessmen and moneylenders - were targeted for ethnic cleansing because they were thought to represent capitalism. They confiscated all cash and paid their people in rations only. To create their agrarian state, they forcibly relocated most of the populations of their cities into agricultural communes. Nearly the entire population was forced to give up whatever trade they had been practicing before to become farmers. Given that these city dwellers had no prior agricultural experience, and that the urban population who did have the necessary agricultural knowledge were unsympathetic or too scared to share their knowledge, this enterprise was doomed to fail.
The people were expected to work nonstop, without adequate rest or food. The Khmer Rouge expected this would make the country produce 3 times the usual agricultural yield, when the majority of the people forced into the fields didn't even know how to produce the usual yield.
Unable to grow enough food, some people attempted to forage for berries, fruits, and wild tubers. These actions were punished by death, for the crime of "private enterprise". People soon stopped doing it, but now they were unable to look for food + unable to grow food because they did not know how. As such, millions starved to death.
Next - malnutrition. The Khmer Rouge's agrarian state was an idealistic notion - not one grounded in science or backed by empirical evidence. For instance, the Khmer Rouge neglected that people have basic nutritional needs that a purely carbohydrate diet cannot sustain - needs like protein. The rearing of livestock was greatly scaled down, and the produce was mostly reserved for the leaders of the Khmer Rouge. Commercial fishing was banned because people were thought to be able to use the boats to escape or communicate with outsiders and bring unwanted capitalist influence into the country. The lack of basic protein needs resulted in mass deaths due to malnutrition.
Finally, illness. The Khmer Rouge did not believe in western medicine, considering it a capitalist influence. They did not have the industry nor knowledge to manufacture it, and thus would have to import it. As such, they relied near-exclusively on traditional medicine. This led to many deaths through illnesses that could have been prevented.
Taking into account the mass starvation, the malnutrition, the illnesses, the ethnic cleansing, the genocide ... the deaths of 1.5 to 3 million should not come as a surprise.
In answer to the second part of your question... the leaders of the bureaucracy and military were loyal members of the Khmer Rouge, and their lower-ranked followers were kept obedient through the deadly political atmosphere brought about by the repeated purges that you mentioned. Their economy was nonexistent - it was meant to be a cashless state.
You didn't need to be forced to join the bureaucracy or military - you would literally beg to be let in. Keep in mind the only other alternative was to be a slave farmer.
And of course, like with all tyrannical socialist regimes, all this - the lack of medicine, the lack of cash, the ban on foreign 'capitalist' luxuries, the malnourished diet, the lack of food - none of this applied to Pol Pot and his closest followers, despite his touted belief in this idea of absolute socialism in the state.
References:
Kiernan, Ben (2008). The Pol Pot regime: Race, Power, and Genocide in Cambodia under the Khmer Rouge. Yale University Press. p. 431.
Khmer Rouge Genocide & the Cham". Cambodian Village Scholars Fund. 15 April 2017.
"Cambodia's brutal Khmer Rouge regime". BBC News. September 19, 2007.
William Branigin, Architect of Genocide Was Unrepentant to the End The Washington Post, April 17, 1998
Thayer, N. (2009-2012). Publications. Soldier of Fortune, [online] Available at: http://natethayer.typepad.com/blog/publications-soldier-of-fortune/
First, it's worth noting that while the figures you stated as estimates of the death toll are correct, they are estimated to have been around a fifth of the total Cambodian population instead of a third. And half of those dead were a result of starvation and disease that resulted as a consequence of the Khmer Rouge's extremist policies of self-sufficiency. In addition to that, a large portion of the other deaths were a result of the brutal labor conditions forced upon the population rather than outright executions.
Now, to elaborate on /u/threechance's first point regarding racial segregation and understanding how the mass executions of the Cambodian genocide were made possible, you have to first understand the core support of the Khmer Rouge.
The kernel around which the regime of Pol Pot was formed were the indigenous tribes of the highlands of Northeastern Cambodia; close to two dozen various ethnic groups known collectively as the Khmer Loeu. These ethnic groups were the original inhabitants of the region, similar to the native tribes of North America and the aborigines of Australia.
These groups have historically been comprised almost entirely of uneducated peasantry, which still to this day have a societal structure built around small villages ruled by a council of elders. They have never fully integrated into larger Cambodian society, have little representation in the Cambodian government, and have always been treated roughly by the ruling class of the country.
For example, Cambodia has never made a treaty with any of the people who comprise the Khmer Loeu. It is also not one of the signatories of the Indigenous and Tribal People Convention of 1989, which was a major international agreement to guarantee certain inalienable rights to the original inhabitants of various countries. The country passed a later law in 2001 giving the tribes rights to the lands they inhabited, but has been repeatedly accused of violating that law and seizing their lands for commercial purposes.
Additionally, Cambodian society has long looked down on the people of the highlands, best demonstrated by the fact the names of some of the tribes have become pejorative terms in the Khmer language. The name of the Phnong tribe has come to mean savage. The name of the Samre tribe has come to mean something akin to a hillbilly.
In the 1960's, thanks to decades of military training given to them by French troops, who viewed the resentful and uneducated tribal people as willing and obedient soldiers to keep the rest of Cambodian society under the thumb of their colonial rulers, thousands of the Khmer Loeu rose in rebellion against the recently formed independent Cambodian government of the lowlands. This resulted in something of an autonomous region for the Khmer Loeu, with the government being unable/unwilling to go into their lands and forcibly put down the rebellion.
It was from this group of people that Pol Pot drew the core of his support, and the ones who comprised his legions of willing executioners against the people of the lowlands, who were the targets of the Khmer Rouge's genocide.
The genocide was not a result of an ideological schism within urban Cambodian society, it was the result of Communist leaders copying the French colonial tactic of harnessing the long simmering resentment of the highland tribes as a weapon against the rest of the Cambodian people, and driving it to the extreme.
Sources:
Kiernan, Ben. The Pol Pot Regime: Race, Power, and Genocide in Cambodia Under the Khmer Rouge, 1975-79. Yale University Press. 2008.
Becker, Elizabeth. When the War Was Over: Cambodia and the Khmer Rouge Revolution. PublicAffairs. 1998.