For much of European history, criminal punishment was extraordinarily brutal by today's standards, with mutilation and death far from uncommon. What led to the decline of violent punishment in the modern era?

There was a combination of direct pressure and indirect cultural shifts over the course of the 18th Century that led to this change. The following quotes are all from Lynn Hunt's Inventing Human Rights: A History, unless otherwise stated.

One early motivating incident happened in 1762, when a French court convicted a French Protestant named Jean Calas "of murdering his son to prevent him from converting to Catholicism." Calas maintained his innocence even under torture, reportedly saying, "Where there is no crime, there cannot be any accomplices." Calas was sentenced to being broken on the wheel, a truly horrific torture in which a person's bones were systematically crushed and mangled before he was left to dangle and die, slowly. "In a secret instruction, the court granted Calas the grace of being strangled to death after two hours of torment, before his body was attached to the wheel," Hunt writes. "Calas died still protesting his innocence."

Things might have stopped there; executions like Calas's were not unheard-of at the time. But his case was picked up by a writer named François-Marie Arouet, which mattered because Arouet is better known by his pen name: Voltaire. In a series of pamphlets, Voltaire first attacked the religious bigotry he thought was behind the prosecution. (Calas's son probably committed suicide.) This largely worked, as over several years the Royal Council reviewed the issue and in a series of decisions eventually acquitted the rest of Calas's family, who were facing the same charges. But as the Calas Affair dragged on, "Voltaire's focus of attention began to shirt, and increasingly the criminal justice system itself, and especially its use of torture and cruelty, came under fire."

Natural compassion makes everyone detest the cruelty of judicial torture, insisted Voltaire, though he himself had not said so earlier... So much did Voltaire's views shift that in 1769 he felt compelled to add an article on "Torture" to his Philosophical dictionary, first published in 1764 and already on the papal Index of Forbidden Books. In the article, Voltaire uses his habitual alteration of ridicule and fulmination to condemn French practices as uncivilized; foreigners judge France by its plays, novels, verses and beautiful actresses without knowing that there is no nation crueler than the French. A civilized nation, Voltaire concludes, can no longer follow "atrocious old customs." What had long seemed acceptable to him and many others now came into doubt.

Another vital campaigner was the Italian Cesare Beccaria, who published Essay on Crimes and Punishments in 1764. This rejected "not only torture and cruel punishment but also — in a move remarkable for the time — the death penalty itself." His influence was widespread. Voltaire read him during the Calas Affair. English jurist William Blackstone read Beccaria and then declared that criminal law should always be "conformable to the dictates of truth and justice, the feelings of humanity, and the indelible rights of mankind" — a very new sentiment.

One critic of Beccaria, Pierre-François Muyart de Vouglans, defended torture and cruel punishments as necessary and justified by common sense. Muyart attacked Beccaria as a bleeding-heart armchair philosopher with no actual experience. Not only was "the sensitivity to pain of the guilty" not a reason to abolish cruel punishments, but it was the whole point. "Precisely because each man identified with what happened to another and because he had a natural horror of pain, it was necessary to prefer, in the choice of punishments, that which was the cruelest for the body of the guilty," Muyart wrote.

But Muyart was fighting against the current of the times. Hunt argues that as much as explicit human rights campaigning, it was broader cultural shifts that removed public support for torture and public executions, by promoting empathy for other humans. She identifies a few things:

The rise of the novel in the first part of the 18th Century drew "readers into identifying with ordinary characters, who were by definition unknown to the reader personally... Through the fictional exchange of letters, in other words, epistolary novels taught their readers nothing less than a new psychology and in the process laid the foundations for a new social and political order." This wasn't an invention of empathy, which is a natural human emotion. Rather, it represented the expansion of empathy to strangers, away from one's immediate tribal group. Defenders of the moral and social establishment said novel-reading "undermined morality, religion, and all principles of social order," and from a certain point of view, they were absolutely right. Thomas Jefferson, in a letter, said that by reading fiction people experience the "strong desire in ourselves of doing charitable and grateful acts" and, Hunt paraphrases, "are disgusted by evil deeds or immoral conduct. Fiction, he insisted, produces the desire for moral emulation even more effectively than reading history."
The rise of portraiture as a popular visual art "encouraged the view that each person was an individual — that is, single, separate, distinctive, and original, and therefore should be depicted as such," compared to an older sense that people represented types or groups rather than individuals.
A change in manners and sensibilities in the 18th Century began to cordon away "violent outbursts of emotion and aggressive behavior" as "socially unacceptable." Especially among elites and the rising middle class, Hall writes, "the threshold of shame lowered, while pressure for self-control rose."

Defecation and urination in public became increasingly repellent. People began to use handkerchiefs rather than blowing their noses into their hands. Spitting, eating out of a common bowl, and sleeping in a bed with a stranger became disgusting or at least unpleasant... These changes in attitudes toward the body were the surface indications of an underlying transformation. They all signaled the advent of the self-enclosed individual, whose boundaries had to be respected in social interaction.

Public executions were thus seen as not merely cruel, but also rude. In 1776, the Morning Post of London complained that the "remorseless multitude behaved with the most inhuman indecency — shouting, laughing, throwing snowballs at each other, particularly at those few who had a proper compassion for the misfortunes of their fellow creatures."

Things began to change. A British visitor to Paris in 1787 watched an execution by breaking on the wheel, and noted the crowd was oddly subdued:

The noise of the multitude was like the hoarse murmur caused by the waves of the sea breaking along a rocky shore: For a moment it subsided, and in an awful silence the multitude beheld the executioner take up an iron bar, and begin the tragedy, by striking his victim on the fore arm.

The crowd, Hunt writes, "no longer felt the emotions the spectacle had been designed to elicit." Meanwhile the rise of individualism was changing criminal theory.

Rather than expiating sin, punishment should be viewed as repaying a "debt" to society, and clearly no payment could be forthcoming from a mutilated body. Where pain had served as the symbol of reparation under the old regime, now pain seemed an obstacle to any meaningful quittance... In the new view, cruel punishment enacted in a public setting constituted an assault on the society rather than a reaffirmation of it. Pain brutalized the individual — and by identification, the spectators — rather than opening the door to salvation through repentance.

As morals shifted and campaigns ramped up, laws began to change. Judicial torture began to be abolished in Europe even before the Calas Affair. England banned it for all but the most serious offenses in the 1689 Bill of Rights. Frederick the Great banned it outright in Prussia in 1754. Judges in Britain's North American colonies began to impose fines for property offenses instead of whipping. In 1789, the revolutionary French National Assembly passed a judicial reform act that "abolished all forms of torture and also the use of a low, humiliating stool (the sellette) for the final interrogation of the accused before his or her judges." In 1791, the Eighth Amendment to the new United States Constitution was ratified, banning "cruel and unusual punishment," while a new French decree embraced prison sentences over corporal punishment for criminals. The guillotine, "invented to make decapitation as painless as possible," was embraced in 1792. Guillotine executions were still conducted as public spectacles, of course, with Louis XVI as a prime example. France didn't actually stop public executions until well into the 20th Century.