There are several important features of the purges that need clarification.

You use the word "cleanses" in your title, which I am going to take to mean "purges." The purges were a particular event under Stalin that occurred between 1933-1938. This is not the equivalent of just generalized violence, oppression, imprisonment under Stalin, so my answer will be aimed at that period in particular. Luckily for us, we have some good documentation about the state's rationale for this. When it comes to how could this happen - we actually have a good idea about it. Let's start in 1933 with the Central Committee:

I. Aims and Purposes of the Purge

The purpose of the Party purge is to achieve a higher ideological standard of Party members, to strengthen the Party organization politically, to secure further confidence in the Party on the part of millions of non-party men and women.

This purpose will be attained, during the purge, in the following manner: (a) by carrying but open and honest self-criticism of the Party members and Party organizations; (b) by checking the activities of each Party cell from the point of view of the execution of the decisions and directions of the Party; (c) by participation in the Party purge of the toiling non-party masses; and (d) by expulsion from the Party of such persons as are not worthy of the highly honorable rank of a party member.

The following elements must be expelled: (1) Alien and hostile elements which have worn their way into the Party by fraudulent means and which remain there in order to corrupt members of the Party;… (3) open and secret breakers of the iron discipline of the Party who do not carry out the decisions of the Party and of the Government, who express doubts in the wisdom of the Party decisions and the plans marked out by the Party, who discredit these decisions and plans by empty talk about their “impossibility” and “futility”; (4) degenerate persons who fall under the influence of capitalist elements, who do not wish to fight class enemies, who do not fight the kulaks, unscrupulous egotists, loafers, thieves and pilferers of public property; (5) careerist, selfish and bureaucratic elements which make use of their membership and their positions in the Soviet administration for their own selfish interests, which have broken away from the masses, which neglect the needs and requirements of the workers and peasants…

(text here)

So, we're talking, from the beginning, about the removal of people from the Communist Party who do not meet the above standards. A few things that may need explanation if you are not familiar with Soviet history. 1) Not everyone was a member of the Communist Party. The Communist Party understood itself as leaders of the revolution and leaders in building socialism. This meant that not everyone was considered suitable for this particular "honor." The purges, then, were aimed particularly at the party.

When you look, then, at the "Great Terror," a term historians use to refer to the particularly bloody years of 1936-1938 but that was not used at the time, the people who had the most to fear were party members, and often high ranking party members.. They were often tried (often in gaudy show trials) for being engaged in terrorist or counter-revolutionary plots, allegations that were contrived and entirely fictional - before being executed (Bukharin et al. being the most famous of those executed as a result of the show-trials). This last point is important because the Bolsheviks were a vanguard party and kept a lot of that mentality even after taking party. In a vanguard party you need professional revolutionaries who are willing to do whatever it is necessary to lead the revolution and anyone who is not on board with that is a liability. I contend that this attitude was very much at the forefront of Stalin's mind when it came to the revolution and when it came to ruling the Soviet Union. (See this post I made a long time ago for more on this.) The threat of counter-revolution is one that Stalin worried about excessively even from the early days of revolutionary activity. He strongly felt that the party had to be kept clean of elements that had the potential to derail the revolution. There certainly is an element of political calculus involved here and I'm not saying that the people killed, exiled, removed from the party and jailed in the purges were actually counter-revolutionary. But that is one way the campaign was framed as being in line with Communism - it was claimed to be a necessary part of maintaining the revolution.

The Bolsheviks had never been shy about using violence to get rid of people that thought were threatening to the Revolution. What is striking about the purges under Stalin is how seemingly arbitrary they became. Although Stalin created the atmosphere for these purges with the high profile show trials mentioned in the previous paragraph, the purges were often carried out by local bureaucrats and party members. Even the state itself realized that this was unsustainable and damaging if allowed to continue unabated. The reason the years I mentioned above end in 1938 is because the party had to a least put some measure of resistance on this process that had otherwise spiraled out of even their centrally held control. So in 1938 the Central Committee decrees:

1. Obkoms, kraikoms, the Central Committees of national Communist parties and all party organizations should resolutely put an end to mass, indiscriminate exclusions from the Party and really adopt an individual, differentiated approach when deciding questions of exclusion from the Party or readmitting excluded members.

(more text here)

At this point you can see that the Central Committee saw a need to end the purges, but that the fault according to them is not Stalin, themselves, or even central power in general. When you read on in that document it becomes apparent that the blame is mainly foisted upon the local Communist Party cadres for excesses in the execution of the purges and not the concept behind the purge.

So, finally, the purges could happen because they were framed as a necessary part of maintaining the Communist Party's revolutionary pedigree. The show trials certainly punctuated the period and created somewhat of a frenzy about terrorism and counter-revolution that enabled local party leaders to also take extreme and often arbitrary measures. Put they did not begin in quite such an arbitrary manner. Again, that doesn't mean they weren't oppressive anyway, just that they nature of the purges changed over time and the frenzy and mania built to a fever pitch in 1938.

Stalin remains the key figure here, and his culpability in the purges is obvious. Nonetheless, the purges weren't just a story of individual excesses among party officials, Stalin or otherwise. It's a striking and somewhat terrifying example of what can happen in a confluence of vanguardist ideology, authoritarianism, an obsession with "in-group" purity and the bureaucracy capable of actually carrying out this kind of large-scale operation.

Again, this has been a discussion about the purges in particular and not about Stalin's repression in general, which is a related, but different topic. If you are interested about general political repression, the gulag, etc., then we can also talk about that in more detail.

This comes from my Russian Government course. Stalin used circular power. He appointed people who supported him which gave him more power allowing him to appoint more people who supported him. The Soviet government at that time made it very easy to do this because even though the Secretary General could be out voted he could appoint people. Ultimately Stalin was surrounded by people who worked for him. This meant he created a cult of personality allowing him to do pretty much anything he wanted, he was more important than the state ideology. The USSR went from authoritarian to totalitarian. Stalin had all of the power and so could kill anyone he wanted because he would always have support. Stalin was brilliant at potilical maneuvering which is a big reason as to how he gained power. He manipulated people quite well. Stalin was crazy, but also quite brilliant as a politician.

I only know the last names of the authors, but try Malia, Kelner, Pipes and Service.

I believe there was also an element of popular movement in Stalins purges. I've posted a bit on that recently, so below is just a c/p of that paragraph:

Stalin's purges begun after a period of New Economic Policy, that seemed to reverse much of revolutionary gains and reintroduce both inequality and exploitation. Also, revolutionary generation of communists was entrenched in mid and high management positions, and that blocked vertical mobility for newly educated generations. Therefore, purges were not only largely supported, but were in many aspects a popular movement. Stalin dealt with whomever he paranoidly thought of as potential opposition, a degree of revolutionary egalitarianism was restored, and vertical mobility once again became possible. I believe that it was the purges that largely provided a strong enough support for Stalin to weather the Second world war. All those who were promoted through purges, felt their social position tied to the figure of Stalin.

(I don't have one single book or an author to support this view of Stalinism and purges, at least not so explicitly. But it kind of comes through works like Fitzpatrick's "Russian Revolution", and especially "Education and Social Mobility in Soviet Union 1921-1934", Neumann's "The Communist Youth League and the Transformation of the Soviet Union 1917-1932", and some others, although I haven't read anyone explicitly calling the Great purge a popular movement.)

Can you explain how it strongly opposes the ideology of the Soviet Union? The Red Terror was a key element to Leninism, just as it was to the Jacobins.

Stalin was not especially ideological.

It helps to bear in mind his rise to power and his role in the Civil War: he was a comparatively unimportant leader fighting in southern Russia and the Ukraine. He was a rural peasant, a redneck, a bogan, a gangster. His power within the party was built on his old mates from his Civil War campaigning: Timoshenko, Voroshilov, Beria etc. While other were taking party and political orthodoxy seriously, Stalin was installing reliable cronies into key positions.

Having established his power, Stalin's paranoid repression fell on leadership figures who could rival him, who were not part of his crony network. He built a cult around himself in the 1930s which created an atmosphere where any disloyalty was automatically treasonous: ideology mattered less than loyalty. The advantage will always lie with individuals who are willing to merely use ideology for advantage, and put it aside to secure power.