Generally there are three reasons given for the decline of the Khmer Empire, though words like 'fall', 'decline' and 'collapse' aren't completely appropriate but we will get to that later.
So, generally speaking, the reasons for the Khmer Empire becoming more like the pre-modern Cambodian nation are:
It is probably more useful to consider these in some kind of consonant whole, as well as noting that there is no single, clear-cut accepted theory of how this happened.
Let's look at some dates to begin with. Most scholars will all cite the same couple of years in a discussion of the Khmer empire: 802 and 1431 AD. These dates, as David Chandler suggests, are not to be thought of as either a beginning or an end to the Khmer civilisation. After all there were people speaking Khmer, doing Khmer things and generally being Khmer for hundreds of years in the region before Jayavarman II began the devaraja cult in what would become the region of Angkor in 802 (the fact that there is a 'second/two' in his name should be a hint). The second date, 1431, the one given for a thai sacking of Angkor, is also commonly but perhaps not certainly given as the date the Khmer abandoned the city and the Khmer Empire was effectively over. This is also not clear, nor certain. There is evidence that some Khmer kings were still active in the region of Yasodharapura until the 1560s - not to mention the fact that Angkor Wat was never left completely abandoned and Buddhist monks remained there up until it was "discovered" by French archaeologists in the colonial period. Anyway, the point of all this is to say that the 600 or so years from 802 to 1431 are not the beginning and end of the Khmer empire, but more delineate the Khmer 'period of greatness'.
So, to the reasons given for the 'transformation' away from said period. The first has to do with the rise of the Thai Kingdoms into a unified and powerful state. These people, who were subjects of the Khmer when the Khmer were at their peak, stopped paying tribute or providing mercenaries and instead began attacking the Khmer regularly. Zhou Daguan's 1296 account speaks of wars with the Thai kingdoms. These kinds of regular sackings of Khmer territory and Angkor itself would be massive drains on both people and resources. Another aspects of this that leads to the 'transformative' aspects of this period is the 'fluid' movement of people across this, for lack of a better word 'border', which caused Thai customs and ideas to move into those of the Khmer. Like aspects of Kingship and spirituality. But, to summarise, basically the Khmer had a new enemy right on their doorstep who was powerful enough for the Khmer to move inland toward what is now Phnom Penh and become more of a trading city near the Mekong.
The second reason, which I feel might be falling out of favour as a 'singular' reason given for the transformation, is the widespread conversion of the Khmer to Theravada Buddhism. A few issues fall out of this, the first is the perception that the adherents to the 'introspective' tradition as opposed to the Hindu one responsible for the more lavish aspects (think massive monuments/constructions) of the Devaraja cult. In a nutshell; maybe you feel less inclined to break your back for the kings new pyramid temple when this theravada buddhism is placing the onus on a personal mindfulness and respect in a prince that gave up his earthly possessions to alleviate your 'suffering'. Thats sort of how the story goes. Although another aspect of that could be the erosion of hindu taxation points in the temples and a higher degree of power moving to local buddhist wats rather than to the resources of the kingdom. Maybe. Again, one can point to the Thai kingdom which was currently kicking the Khmer's ass and point out that they were all Theravada Buddhist - so were the Burmese Kings who were controlling that region. So, like I said, it is not usually given as the sole reason for the 'collapse' anymore, but it should be thrown in there for balance.
The third reason is the one I find most convincing and is the most recent one added to the list. An economic collapse of the megacity that Angkor/Yasodharapura had become, and this collapse being caused by an environmental disaster. So archaeologists using a technology called LIDAR have in recent years mapped out the city of Angkor to an impressive amount of detail and basically figured out it was the largest pre-industrial city in the world. Looking at maybe 750,000 - 1,000,000 people living there. The success of this city, and the empire itself, was built upon a complex set of causeways and artificial reservoirs that allowed the Khmer to catch heaps and heaps of rain during the monsoon season and survive comfortably into the dry season. In fact they could produce 3/4 harvests a year, the surplus of which can be used to fuel conquests, or more building projects - basically just keep everyone happy.
Climate scientists have discovered that there may have been extensive periods of drought in the 13th/14th century, which would have seriously messed up Angkor's delicate ecosystem of artificial lakes and causeways. What may have compounded this problem was the amount of excavations they were carrying out in order to build a larger and larger city, e.g deforestation leading to increased silt runoff which then ruined your artificial rivers. A side suggestion here is that the widespread stagnation of these areas of water could have led to malaria outbreaks.
basically, if mother nature shoved a stick into the system of water management that the Khmers had built up for hundreds of years, it was going to cause a massive problem. Think of it as the equivalent of your country's roads or trainlines being completely ruined for a few years. Everything would grind to a halt.
So lets add these reasons together. You've got constant fighting with a new enemy that keeps taking your people as slaves when they win, everyone is a Buddhist now so they aren't giving the previous amounts of respect to the King or his system of taxation, and meanwhile you've got droughts screwing up the insanely delicate and complex system of water networks you've literally based your whole empire on.
So, let's move the whole damn city down the road where we won't have to deal with half of these problems. Unfortunately doing that means you are going to be giving up on some of the aspects of your civilisation that people in the future will remember as 'great', like all those nice and super impressive bits of architecture and design or those areas of the map that used to be your territory. In this sense we see the 'transformation' of the Khmer Empire into the Khmer civilisation which would eventually become plain old Cambodia.
I touch on the rise and fall of the khmer empire in episodes 2/3 of my podcast: In The Shadows Of Utopia
Sources for this include David Chandler's A History of Cambodia and publications of the Greater Angkor Project by people like Damian Evans, Roland Fletcher as well as interviews I have conducted for the podcast.