I read that after WW1, there was a regency for emperor Charles I, but he was prevented from taking the throne by the regent, twice. Why did this peculiar situation happen? Why didn't the Hungarian elect another king or do away with the monarchy entirely if the (rightful?) king is not allowed to take it?
TL;DR There was no concensus on who should be the king, the political landscape was split roughly in two factions, one that wanted Charles back and one that wanted the right to freely elect a king; as sort of a compromise, after Charles failed to gain the throne the second time, this question was shelved indefinitely for the sake of unity. There was little desire to get rid of the monarchy as following the fall of Béla Kun's marxist-leninist republican dictatorship, republicanism was associated with all sorts of issues and the post-Kun leadership was generally very conservative and reactionary.
Let's adress the issue of the monarchy itself - first off it has to be said that Hungary was, for a relatively brief time that is, a republic immediately after WWI albeit a very chaotic one. Mihály Károlyi, an aristocrat and a liberal politician, led the so-called Aster Revolution that forced the Emperor (or for Hungary specifically King) Charles to appoint Károlyi as the Prime Minister of Hungary on 31 October 1918 and on 13 November Charles issued a similar proclamation that was already given to Cisleithania two days prior, that is that he won't be involved in the Hungarian affairs of state, allowing Károlyi to abolish the Monarchy and proclaim the Hungarian People's Republic on 16 November.
However this republic was rather short-lived as in March 1919, communists and social democrats merged together under the marxist-leninist Béla Kun, seized power in the republic and proclaimed the Hungarian Republic of the Councils (or the Hungarian Soviet Republic if you will). It started the 'red terror' brutally persecuting any potential threat to this new regime and prepared for a war against its neighbours trying to hold on to the lost territories.
Kun's regime however lost the war (Kun himself fled to the Bolshevik Russia where he would be later executed during the Stalinist purges) against Hungary's neighbours and Budapest was occupied by the Romanian forces. Meanwhile in the city of Szeged an anti-communist and conservative/monarchist/nationalist opposition was formed, chief among them being Admiral Miklós Horthy with his army (the so-called 'National Army'), Gyula Károlyi (Cousin of Mihály), István Bethlen and Gyula Gömbös with his right-wing paramilitary organization known under the abbreviation MOVE (Magyar Országos Véderő Egylet; Hungarian National Defence Association in English). The National Army as well as MOVE started its own 'White Terror' against communists, Socialists, Liberals as well as Jews and Freemasons among others.
It has to be said that this was not the only opposition in Kun's Hungary, however it was the most prominent.
After the communists were defeated, there was an attempt by a socialist Gyula Peidl to reinstate the pre-communist republic, however István Friedrich, a factory owner and a conservative politician who was arrested and sentenced to death (though his sentence was commuted) during the Red Terror, he led his own anti-communist and conservative faction in Budapest called the 'White House', in early August 1919 overthrew it, became the prime minister and proclaimed the previous republican regimes as illegitimate. He reinstated the Monarchy (with all of the old laws, including the Pragmatic Sanction of 1723) and appointed his Habsburg ally, Archduke Joseph August, as regent. The Archduke did not last long due to the pressure of Entente and its Allies in Central and Eastern Europe and was replaced by Horthy.
Now why was the monarchy reinstated? For the 'white' forces in Hungary, it was republicanism, or more specifically the distain for the historic Hungarian royal institutions, that caused the chaos during the Károlyi and Kun's regimes (in this way, monarchy was also seen as an anti-thesis to these 'hated' regimes). Moreover for the conservatives the Hungarian Holy Crown symbolized the unity and integrity of the monarchy and its territories which in their eyes were broken the moment Károlyi proclaimed the republic.
Now came up the question of who should actually sit on the throne. The politicians were split on this issue roughly in two camps - the legitimists, who argued for upholding the Pragmatic Sanction which meant that they supported the claim of Charles, and the free electors who wanted to see the Pragmatic Sanction gone and argued for a right for the Hungarian people to freely elect the new Hungarian king be that a Habsburg or not.
The legitimists were generally speaking from prominent pre-1918 aristocrats and catholics, the most influential among them being Gyula Andrássy and most of the Christian National Union Party. The free electors were generally protestants, most of the factions inside the National Smallholders and Agrarian Workers Party were generally leaning towards the free election however with various degrees of flexibility, the aforementioned Gyula Gömbös, which would later in the 30s become the prime minister of Hungary, was among the most prominent and radical free elector.
These factions were difficult to reconcile, which paralyzed the Hungarian parliament, however the ambitious István Bethlen, who became the Prime Minister in April 1921, was determined to do it - from his perspective Hungary has suffered from a lack of national unity and the only remedy according to him was to unite all of the conservative factions, that is most of the Smallholder Party and the Christian Nationals, into one party.
He was ironically helped by Charles when he attempted to gain throne the second time in October 1921 - Charles flew to Denésfa and from there he travelled to Sopron where he mustered some legitimist troops, he also announced his governmental cabinet in which prominent legitimists were seated, however he did so without consulting them first. Andrássy, one of the appointees, had the misfortune of actually being in Denésfa at the time Charles arrived (doubly so because he reached an agreement with Bethlen few days prior), this was however pure coincidence and rather unfortunate timing as he was really attending a christening. Needles to say Charles was stopped by a force of students and right-wing radicals led by Gömbös (with the approval of Horthy who disliked Charles) in Budaörs, right outside Budapest. Charles himself was after a clash of arms arrested and expelled from the country for attempting a coup d'etat.
The fact that Andrássy was both present in Denésfa and appointed into Charles' government meant that Andrássy and several others were arrested and the legitimist position was greatly weakened. Bethlen used this and successfully pressured the legitimists to give up on the Pragmatic Sanction of 1723, meaning that a road to free election was possible. However, Bethlen, probably afraid of pushing the legitimists too hard, went the way of compromise, on one hand annulling the Pragmatic sanction and on the other refusing to rule out the idea of a Habsburg king, so the way for Charles was still left open. The question was thus shelved and was supposed to be resolved at a 'suitable time'. This move managed to appease both the legitimists and the free electors, Bethlen was obviously satisfied with this resolution and did not want to press the issue further.
Sources:
BODÓ, Béla, The White Terror: Antisemitic and Political Violence in Hungary, 1919 - 1921, 2019
LORMAN, Thomas, Counter-Revolutionary Hungary, 1920–1925: István Bethlen and the Politics of Consolidation, 2006