It's considered a genocide because it targeted specific ethnic and religious groups. Raphael Lemkin's strict definition of genocide is as follows:

[G]enocide means any of the following acts committed with intent to destroy, in whole or in part, a national, ethnical, racial or religious group, as such: (a) Killing members of the group; (b) Causing serious bodily or mental harm to members of the group; (c) Deliberately inflicting on the group conditions of life calculated to bring about its physical destruction in whole or in part; (d) Imposing measures intended to prevent births within the group; (e) Forcibly transferring children of the group to another group.

So with that definition in mind, it's clear that the Cambodian genocide fits right into that framework. Dr. Gregory Stanton, in his conference paper The Cambodian Genocide and International Law notes that, "The Khmer Rouge singled out certain religious and ethnic groups for elimination." One of his examples is a Khmer Rouge order which quite clearly calls for the elimination of the Cham people, stating, "A Khmer Rouge order stated that henceforth “The Cham nation no longer exists on Kampuchean soil belonging to the Khmers"." The Cham were massacred extensively during their genocide and their children were taken and given to Khmer villages to raise, ticking A, B, C, and E of Lemkin's definition. Unfortunately though, the Cham were not the only group targeted. Stanton also notes that Buddhist monks, Christians, Vietnamese minorities, and Chinese minorities were all intentionally targeted as part of the killings. This was in addition to Khmer people who were being killed as well.

Professor Ben Kiernan, Director of the Genocide Studies Program at Yale, cites three distinct categories of victims for the genocide in his paper The Cambodian Genocide 1975-1979. These are:

A. Religious groups, namely Buddhists and especially Buddhist monks. According to orders translated by Kiernan, the Khmer Rouge wanted to wipe out all religion from the face of Cambodia, so they went about targeting Buddhist monasteries and places of worship. Of the 70,000 monks living in Cambodia at the time, at best only around 2,000 survived the killings. Christians were also killed en masse, including destroying the Phnom Penh Cathedral. Stanton notes that only one Christian pastor in the entire country survived the genocide.

B. Minority ethnic groups, primarily Chinese minorities, Vietnamese minorities, and the Cham. The Pol Pot regime arleady actively oppressed these minorities, so it was no suprise that they were targeted in the killings. According to Kiernan, of the 425,000 ethnic Chinese living in Cambodia in 1975, less than half would survive. Similarly, Kiernan states that, "In research conducted in Cambodia since 1979 it has not been possible to find a Vietnamese resident who survived the Pol Pot years there." Kiernan specifically calls the killing of the Vietnamese a "campaign of systemic racial extermination" but admits that in some cases the killings of Chinese minorities was more a matter of the fact they lived in an urban environment, making them targets for the anti-urban purges. However, there's no controversy about the targeting of the Cham, who, as both Kiernan and Stanton note, were exposed to systemic religious and ethnic persecution and widespread massacres because they were Cham. Other ethnicities were also killed. Per Kiernan

The toll goes on. The Thai minority of 20,000 was reportedly reduced to about 8,000. Of the 1,800 families of the Lao ethnic minority, only 800 families survived. Of the 2,000 members of the Kola minority, ‘no trace… has been found’.

The final group of victims Kiernan cites is probably the most controversial in terms of defining a genocide, and that's the specific targeting of the majority national ethnic group, the Khmer. I have seen Cambodia referred to as the only case of auto-genocide, i.e. a situation in which those carrying out the genocide target their own people for extermination. I tend to agree with this assessment, though it is certainly not as cut and dry in terms of genocide as the killings of religious groups and ethnic minorities. That being said, the argument for the targeting of Khmer people is quite strong. There's not doubt that the regime was specifically purging Khmer peoples, whom they essentially grouped into acceptable and unacceptable ethnic subclasses (with the unacceptable ones getting killed or worked to death). Kiernan notes that undesirable Khmer were "marked as easterners by being forced to wear a blue scarf... and later eliminated en masse." That sounds like a genocide to me, even if the perpetrators are artificially creating the groups they target, but this is probably the part that fits least with Lemkin's strict definition of genocide.

Regardless, by dint of the religious and ethnically focused killings alone the Cambodian genocide is most certainly a genocide per Lemkin's definition. Hence at least partially why it is referred to as such.

Sources

Ben Kiernan, The Cambodian Genocide, 1975-1979 https://www.niod.nl/sites/niod.nl/files/Cambodian%20genocide.pdf

Gregory Stanton, The Cambodian Genocide and International Law http://www.genocidewatch.org/images/AboutGen_THE_CAMBODIAN_GENOCIDE_AND_INTERNATIONAL_LAW.pdf

Yale University Genocide Studies Program, Chronology of Cambodian Events Since 1950 http://gsp.yale.edu/case-studies/cambodian-genocide-program/publications/chronology-cambodian-events-1950