It was rather often that the Communist Parties that gained power weren't popular parties in modern and established economies choosing equality over the methods that developed them. Rather they were Vanguard Parties. The concept was something of a significant departure from earlier Communist Thought. Rather than socialist revolution coming from establish democracies and modern economies where the middle class overthrew the elite it was coming from places that hadn't yet gone through that stage. Which upended a lot of the assumptions of Communist Theory.

So, what is a Vanguard Party? Well, it's a smaller cadre of professionals, an elite if you will, who attempted to take control of a country that isn't entirely ready for end state communism and steer it to those ends. In the case of Russia it was a small group of hardcore professional revolutionaries and the people they trained. Really, this is a major point where Leninism diverges from Marxism. Given that the Soviets were the primary backer of all later Communist movements and given that they were the ones giving out training, guns, and money those willing to incorporate the Soviet ways of doing things were able to crowd out native communist movements.

So, Vanguard Parties were what happened. So, how did they end up very centralized rather than very decentralized? Well a bit one was bit of mental slight of hand. You see, the Vanguard destroys anything that is not the workers. The workers make up the nation. Therefore the nation is the workers. If the workers and the nation are the same thing then the nation owning the land is the same thing, right? But, in practice, it's this elite, enlightened vanguard that ends up owning the thing. After all, the workers aren't ready for direct control as they haven't gone through the whole process described by Marx's theories yet, going straight from rule of autocratic kings to the rule of "the workers" (and by that, they meant the Vanguard).

A lot of the totalitarianism was theoretically justified by turning the workers into Homo Soveitcus, a fully modern and fully communist entity that would be ready for the socialist utopia that was inevitable in coming. This was understood as being as removing anything that was getting in the way. After all, if people don't have a choice but be that way then they will be that way... only, a lot of the Communal (read: State-Owned) enterprises and farms didn't work out as well. Soviet Agriculture, for example, was a disaster. The communal farms were never really able to produce as well as their western counterparts. This isn't an indictment of Russian Peasant, as the roughly 3% of farm land that was private plots produced about half of the nation's meat and fruit. The Soviets themselves figured that it was a lack of machinery that could take advantage of the larger scale of farms and a lack of infrastructure to distribute what was produced. At the same time western farms were consolidating in a similar manner, after all. In the United States factory farming was developing to get the most out of the various equipment being developed. It was hard for a small family farm to make use of a giant thresher, but with enough land a giant thresher can thresh much more cheaply than can be done by hand or even by smaller machines. So, the flaw has to be something other than just "large, centrally managed farms are inefficient" going on there.

There were several problems, so in addition to those that the Soviet leadership identified, there were things called agent-principal problems. In their dismantlement of Labor Unions they pointed out that the leaders of Unions often acted in their own interests despite their power coming from the workers. Basically, an agent-principal problem is any case where a person is empowered to make decisions for a larger group but acts in favor of their own values and goals and neglects those of the people they should be acting on the behalf of. Despite this accusation being leveled against the less political labor unions in the West, it's especially clear that it's what happened to these Vanguard Parties. If they were successful they became a new Aristocracy whose reason for existence was to bring workers to the point where they could own and rule in their own right and gathered all power to themselves to have the ability to make that happen. Though, they ultimately failed to make a great deal of progress because once they had all the power making decisions that would entail the loss of that power over time wasn't exactly the decision made. Once the Soviet Union blazed that path and richly rewarded anyone who followed it became the dominant mode of Communist expression throughout the century.

You may be interested in an answer I gave here discussing the ideology of Soviet leadership towards classless/stateless societies. In short, though, Marxism only advocates decentralization during communism, not socialism, which is what "communist countries" were; they were socialist states trying to advance forward into communism.

As for authoritarianism, Marx and Engels encouraged such means. Engels wrote in 1872:

[Anti-authoritarians] demand that the first act of the social revolution shall be the abolition of authority. Have these gentlemen ever seen a revolution? A revolution is certainly the most authoritarian thing there is; it is the act whereby one part of the population imposes its will upon the other part by means of rifles, bayonets and cannon — authoritarian means, if such there be at all; and if the victorious party does not want to have fought in vain, it must maintain this rule by means of the terror which its arms inspire in the reactionists. Would the Paris Commune have lasted a single day if it had not made use of this authority of the armed people against the bourgeois? Should we not, on the contrary, reproach it for not having used it freely enough? Therefore, either one of two things: either the anti-authoritarians don't know what they're talking about, in which case they are creating nothing but confusion; or they do know, and in that case they are betraying the movement of the proletariat. In either case they serve the reaction.

Might be better suited to /r/asksocialscience.

I think you are almost hinting at the answer in your question. It is because the parties and states that called themselves communist or socialist were far from being so. I am going to stick to the Soviet Union while answering your question as it is the one I am most familiar with, perhaps the most brutal society and the one I suspect most here are most familiar with too.

Of course, "communist" or "socialist" is a term, like many in politics, that has been so eviscerated of any meaning that it is difficult to use, but from its very earliest days the core of socialism or communism was democratic employee control and ownership of the workplace, or what Marx called "workers' control of the means of production." That concept was entirely lacking in the Soviet Union, where control and decisions were made from the top down and there was little input from workers' councils. Stalin increased this by introducing Edinonachalie or one-man management, where one official ruled the workplace and was responsible for virtually all decisions. These bosses were given presitge, power and very high wages, so much so that Stalinist statisticians were embarrassed that the ratio of workers' to managers' pay was higher in the Soviet Union in the 1930s than in the United States.

This top-down structure was mirrored in the parties themselves, which, as /u/a_soporific has mentioned, were vanguardist, meaning they saw themselves as the leaders of the state and expected obedience from subordinates. This is, needless to say, a highly authoritarian structure.

Therefore, we should address why both these governments and Western governments continued to call them communist or socialist when they diverged so wildly from what was understood to be the core socialist principle. In the USSR, the government continued to call itself communist because of the very good light in which people (particularly working-class people) saw the concept. In the US, it was done for the absolute opposition reason: in order to defame the concept of socialism entirely, to associate it with authoritarian regimes.

We must also address the context in which communist governments like the USSR came to power. The Bolsheviks seized power after a devastating 3 year war which wrecked the society and the economy, to the point where starvation was widespread in Petrograd. There then followed an intense war where the British Empire, the United States, the French Empire, Japan, as well as host of other countries invaded Russia in order to "strangle bolshevism in its cradle", as Chuchill said. This led to further destruction of the country and to the new government developing under an all-out fight to the death against people inside and outside Russia who wished to destroy them. The society had not recovered when fascism rose around Russia's borders, in Germany, Japan, Poland and other states. one of the key messages of fascism was to destroy communism, and so Russia was dragged into preparing for another inter-contenental war of annihilation. Often historians portray the Soviets as paranoid, but their paranoia turned out to be justified. Stalin said in 1931 "We are fifty or a hundred years behind the advanced countries, either we make good that gap in 10 years or they destroy us." Germany invaded 10 years later.

It is in this context that the decision of the communist governments were made: under threat of complete annihilation. They were seeming permanently at war and permanently in danger.

We can use Marx's own framework of historical materialism to answer the question.

In Marx’s theory of history, called historical materialism, society moves forward to new states, cultures, economic systems, etc. through a conflict between economic classes. A wealthy exploiting class, possessing property while the rest of the society use it but do not own it, is overthrown by the exploited class, creating a completely different socio-economic system. Marx’s definition of class is not based on things like income level, but whether or not one has any ownership of the means of production (factories, restaurants, stores, etc. and all other businesses). In one very critical phase of Marx’s historical timeline, capitalism has the function of modernizing and industrializing a society. This consequently creates a very huge sum total of wealth in a society despite the fact that the majority of the wealth is owned only by a very small section of the business-owning population. Socialism would follow capitalism as Marx argued, and it would improve upon the society by having the entire working class democratically control businesses allowing for a more equitable distribution of wealth (although not a perfect equality and not without material incentives for us to still pursue).

Milovan Djilas, a Marxist critic of Communism from Yugoslavia, published The New Class: An Analysis of the Communist System in 1957. Djilas used Marx’s theory and progression of history to underline a fundamental problem: most or all of the societies that underwent leftist revolutions were not fully capitalist and even to a great extent feudalistic like Russia. As a result, Marx had been wrong on a key point of predicting that the advanced capitalist regions like North America or Western Europe were to be the first movers to socialism. In many of these new Communist countries the leaders themselves understood that their society was too backward and impoverished for socialism and so they developed their nation through an economic system called state capitalism – a term many believe best describes the nations we falsely call “Communist” whether past or present. The attempt to rush through this phase of Marxist history had failed as Djilas argued, and a new bureaucratic class had formed signifying a lack of democracy and socialism.

Djilas goes in great detail to solidify his description of the “new class,” which involves them doing everything from using Marxism as a form of religious dogma to inflating their personal wages like CEOs in capitalist countries. As mentioned, the new class had formed because these societies based themselves on state capitalism to industrialize and modernize themselves. One critical aspect of this involved highly centralized planning to industrialize a nation more quickly than a market could, especially during Stalin’s rule and regardless of how much death and suffering was involved. Djilas explains that such intense centralization and capitalist behavior in the society gave birth to a bureaucratic and oligarchic class unaccountable to the vast majority of citizens living in Communist nations, a ruling class just like the slaveholders in ancient Rome, the land barons in medieval Europe, or capitalists in the United States.

Would anyone care to comment on Hayek's argument in Road to Serfdom?

I recently read it. It seems to offer an answer to this question, but I'd like to know how accurate professional historians think it is.