The Khmer Rouge held that the Mekong Delta region in southern Vietnam was their rightful territory. Historically, the territory was part of Cambodia until the 17th century. A Cambodian King allowed Vietnamese to settle there, and in the 18th century, Chinese-Vietnamese forces under Mạc Cửu conquered the territory in the south entirely, placing it under Vietnamese administration.

This became officially delineated by the French colonial forces, who also handled some of the islands between Cambodia/Vietnam in the Gulf of Thailand. In 1939, Jules Brévié drew a line and gave everything north to Cambodia and everything south to Vietnam, but Cambodia disputed this. The most notable of these islands even today is perhaps the Phú Quốc/Koh Tral islands, with many Cambodians refusing to recognize the occupation of these islands by the Vietnamese government, including the CNRP (the largest opposition party). Sam Rainsy in particular brings up Kampuchea Krom a lot (the Vietnamese Mekong Delta + the islands), probably because Hun Sen is seen as pro-Vietnamese, but modern Khmer irredentist beliefs aren't particularly relevant to the question except to show that this is a relatively common undercurrent in Khmer politics.

The Khmer Rouge was one of these Khmer parties that did not recognize Vietnamese control of these islands or the wider Mekong Delta. On May 1st, 1975, a group of Khmer Rouge soldiers invaded and occupied the island, but were quickly repelled by the Vietnamese. This was the first aggressive move in the conflict, but not the last. This prompted a series of broader border skirmishes across the Vietnamese-Cambodian border, and for a variety of reasons, a struggle within the Khmer Rouge where pro-Vietnamese or Vietnamese trained cadres were purged in 1976. Cambodia at the time was pro-China, while Vietnam was pro-USSR, and the Sino-Soviet split was ongoing, likely influencing this, but there's a lot of historical animosity, and Pol Pot at the time was trying to reestablish some nationalist vision of Angkor that did not allow for Vietnamese occupation of historical Khmer territory.

Whatever the reason, relations between the two countries got worse after the purge, and by the late 1970s, Cambodia was actively preparing for a war with Vietnam. On April 30th, 1977 tensions skyrocketed when a Khmer Rouge force crossed the border and massacred hundreds of Vietnamese civilians. The Khmer Rouge government demanded the withdrawal of Vietnamese troops from disputed areas and the establishment of a DMZ, which Vietnam ignored. In September, this culminated with the invasion of Vietnam by six Khmer divisions, and thousands of civilian massacres. Vietnam was no longer willing to tolerate this, and invaded Cambodia in 1977/78, but they ultimately withdrew from the territory after failing to get the Khmer Rouge to recognize their claims. This prompted a change in thinking in the Vietnamese government, as they realized that only through regime change would they successfully settle the border dispute, so this early withdrawal in January of 1978 (despite being only 38km from Phnom Penh) was to help plan and prepare for a more sustained occupation later on.

This withdrawal emboldened Khmer Rouge leaders, who began to step up their border incursions. Ultimately they were repulsed, and with a more suitable invasion force, Vietnam returned to finish the occupation in the summer of 1978. They established a new government, and set about recording the Khmer Rouge atrocities as a way to justify the regime change (and it did!), but they wouldn't have pursued it if the Khmer Rouge had been willing to sign the border agreement. In the early 80s they signed a new border agreement with the new government led by Hun Sen (still PM of Cambodia today!).

The Western world was split on the invasion. The most aggrieved parties were China and the USA, who had repaired their relationship in the 1970s. Some in Europe supported the invasion and recognized the new government, but China would launch a counter-invasion of Vietnam in 1979, and the USA would recognize the Khmer Rouge as the legitimate government of Cambodia until 1991 (the Khmer Rouge also sat a delegate at the U.N., not the Phnom Penh government). Someone else might be able to go into more detail but its basic Cold War politics. The USSR supported Vietnam, so NATO obviously couldn't.

Communism in SE Asian largely started with Vietnam. Communist parties initially sought independence from France, and were hugely influenced by Ho Chin Minh and the Vietnamese Communist Party.

From after WWII to just before the Vietnam War, the Vietnamese and Cambodian Communists worked incredibly closely with each other, initially beginning as anti-colonial forces. In both the first Indochina War and the Vietnam War, Vietnamese troops used Cambodia for supply lines and to hide troops, and the Cambodian Communist Party had many Vietnamese members who helped to guide its foundation. The term "Khmer Viet Minh" was used to describe many of the early Cambodian communists, as significant numbers were trained in Vietnam and worked closely with the Vietnamese Viet Minh. However, the Vietnamese Communists envisioned a united Indochina largely dominated by Vietnam, while many Cambodian Communists saw this relationship as too subservient to Vietnam.

After the Geneva Conference in 1954, the Viet Minh and all Vietnamese communists had to pull out of Cambodia, in accordance with the conference. This, combined with actions of the monarchy that decimated the moderate Communists, allowed more radical elements of the Communist Party under Pol Pot to take control. It was during the following years, where Vietnam was involved in the Vietnam War and Cambodia devolved into civil war, that breakdowns began to occur between the two Communist parties. Vietnam wanted the Cambodians communists to stay peaceful and win through elections; they feared that an armed conflict would destabilize the region too much, and they were embroiled in a war and thus wanted stability from their allies. The monarchy's anti-Americanism was seen as good enough by the Vietnamese, and they refused Pol Pot's requests for weapons.

The Cambodian civil war deserves separate reading on its own, but long story short, the prince enacts policies to please conservatives that leads to the radicalization of the Communists, Lon Nol stages a coup, and the American-backed Nol government fights the Maoist Communists under Pol Pot, a war that the Communists win. Significantly, during the war, Lon Nol led massive anti-Vietnamese purges and massacres that riled up Cambodian nationalism, and Pol Pot continued these as they served his interests. This obviously strained things between Cambodia, now firmly in the Chinese/Maoist Communist camp, and Vietnam, which was in its own camp and sought to exert its influence. Pol Pot's anti-Vietnamese rhetoric culminated in incursions on Vietnamese land and the massacre of hundreds of civilians, further straining tensions.

In the buildup to the Cambodia-Vietnam War, the Vietnamese saw anti-Vietnamese sentiments and a throwing away of "true" Marxism-Leninism in Cambodia, while Cambodia saw a growing Vietnam attempting to absorb Laos and Cambodia into itself. The atrocities committed by Cambodia on ethnic Vietnamese was a major reason for the invasion, as well as the desire to replace the heavily anti-Vietnam Khmer Rouge with a more pro-Vietnam party that would help foster their dreams of a Vietnam-dominated Communist Indochina. The West didn't support it because it was an infraction on Cambodian sovereignty by invading Vietnamese troops; China didn't support it because the Khmer Rouge was squarely in their basket; and the USSR did support it because the Vietnamese Communists were more closely aligned with the Soviets than the Cambodian Communists were.

In relation to the question about 'why did they (Vietnam) invade?', I've answered some questions in the past that tackle this area particularly, as well as the question of altruism and intent. Was this to stop a ‘genocide’ as you say? Or is the idea of Vietnam as humanitarian interveners a little less simple in reality? For an answer that goes slightly beyond what is presented here, as well as sourced, you could check out:

​

What made Vietnam invade Cambodia? Why did they suddenly care about human rights violations?