Why not enjail or kill them arbitrarily like it was done in the contemporary right-wing dictatorships and in the colonies?
They did certainly not have to keep up appearances for the press or the judiciary branch since both were under tight control of the selfsame security apparatus that did all of the torturing-some-folks.
So why did they go through the hassle of getting, sometimes literally blood-stained, confessions? Why not forego all these formalities or write a confession for the prisoner on your own? It should have been painfully obvious it is a witch hunt to everyone involved in this procedure and yet they did everything by the letter of the law, which happened to allow for some extended interrogation techniques due to the state of "capitalist encirclement".
Are there any books which attempt to explain this peculiar Kafkian side of Stalin's dictatorship? Are there any philosophers attempting to draw lessons from it for the other countries with a formal rule-of-law?
This question is irritating me as I have definitely read a detailed examination of this and I can't remember where. It may have been Halfin's Stalinist Confessions: Messianism and Terror at the Leningrad Communist University.
For the Stalinist purges and, indeed arguably the revolution itself, a great deal of stock was placed in being "right". Seen through the mechanism of dialectic materialism, the socialist revolution is an expression of "History with a capital aitch". So too is the dictatorship of the proletariat and the idea of revolutionary violence.
The Soviet organs of repression needed the people to believe that they were always right. They needed to convince themselves of that. The only way to justify the trials and the purges was to convince everyone that they always had the guilty party, even if that was forced by confession. This also had the effect of encouraging the population to remain obedient and calm - as they would trust that only the guilty could be sanctioned.
It is notable from a lot of the literature, especially Orlando Figes' work, that it was common for victims of the purges to say things like "Obviously I was innocent, but all the others were guilty".
The contrast with, for example Nazi Germany lies at the heart of what these regimes were trying to achieve. The victims of the Holocaust were condemned for their natural traits - Jewishness, Homosexuality etc. That did not need to be proved to a court as it was so evident, though do remember the regime developed various bureaucracy for identifying victims (different grades of Jewishness, measuring physical features). The Soviet system was different as it was repressing citizens, and that needed to be justified.
There was also a form of redemptive belief in the process. By providing a confession, the prisoner was accepting that the state was right. They were even proving their loyalty by confessing to the "alternative facts" of revolutionary justice - see further 1984 - all part of the grand narrative and self-justification of the Stalinist state.
I hope this answers your question. A lot of this is half-remembered from something I read a while ago. I'd be happy to try to clarify if you have follow up questions.
