I've discussed the medieval concept of "proof"--especially of naughty deeds--in an earlier answer that I've edited somewhat for here, if you're interested.
~~
It's true that no one would ever use the term "medieval surveillance state." However, the problem with that phrase is state, not surveillance.
You might be familiar, in media discussions, of the idea that we live (or in these accounts, lived) in a narrow band of history where there could be an expectation of privacy in the Western world. The typical image invoked is the village small enough that everyone knows everyone else's business. But that's still a fairly recent view.
Rewinding to the Middle Ages, we meet a concept called fama. This is a Latin word that means reputation or word on the street or rumor, some combination of those--and in medieval courts, fama was a legal principle with concrete implications.
Bad fama was used to discredit witnesses or reject their testimony altogether. According to 13th century French legal texts, in a lawsuit between someone with bonne renomee and someone with mals renome, the first person would receive the benefit of the doubt automatically. In some cases, bad fama would cause a person's lawsuit to be dismissed out of hand, or permit them to push for charges of fraud.
F. R. P. Akehurst citing civil jurist Philippe de Beaumanoir gives this exemplum of the power of fama--and who had control over it:
An innkeeper with a good reputation could avoid charges of having stolen property from his guests, but if his reputation were not very good he would be the most likely suspect, even if there were signs of forced entry and broken chests. In such a case, the reputation of the innkeeper would be determined by a judicial inquiry.
The only kind of evidence such an inquiry would turn up would be oral: what people said about a person could make or break him. The inquiry also delved into what other people thought of a person, what they remembered of him.
While Beaumanoir is writing a prescriptive text, Akehurst compares the procedures listed favorably in terms of reflecting contemporary practice. So we should take seriously what Beaumanoir is saying here: forensic or physical evidence did not determine the case; other people's opinions of a person close to the crime did.
Gossip made reality.
And in cases of blackmail, this could apply as easily to something invented as to something that had actually happened.
There is also, for the late Middle Ages into the early modern era, the question of the sacrament of confession. This has a vast and contentious historiography, so in advance, I want to be clear that we have to distinguish between "the population at large" and "some individuals here and there"--that is, not everyone has the same experience or depth of exposure/intensity/care.
In 1215, the Fourth Lateran Council very famously (well, okay, very famously to medievalists, which is not really very famously at all) decreed that all Christians of both sexes must confess their sins to their parish priest once a year. The actual point wasn't confession itself, of course--it was that lay Christians must receive the Eucharist once a year, and confession was necessary to cleanse one's soul before what was central enough to be just called "the sacrament."
In practice, however, the confession-Eucharist connection amounted to a strong focus on both sacraments in religious instruction: the Eucharist, that it was the genuine body and blood of Christ and reception was necessary for salvation; confession, what sins were and what was moral behavior and the necessary contrition-confession-penance triad. Oh, yeah, and that really you needed to confess to a priest and receive sacramental absolution; just shouting at the sky was insufficient for salvation purposes.
Now, this doesn't mean that in 1216, every Christian in the medieval West was confessing their sins on Palm Sunday just like that. However, participation ramped up over time. Scholars pretty much universally agree that by the fifteenth century, Christian Europeans were indeed confessing their sins once a year.
More relevant is whether people would really tell their naughty naughty deeds to the priest. And the answer is...well, we can't know for sure, but there is immense evidence that points to people taking confession, absolution, and penance very, very seriously as a necessary gateway to heaven.
The most important example is the increasing availability of confession. Not just the once a year routine. Rather, by the fifteenth century there were dioceses mandating confession more than once a year. Others report it was offered more frequently to certain groups ("women and students" being my favorite example). This is a question of demand.
Late medieval people also won the right to choose their own confessor, despite Lateran IV's mandate of confessing to the parish priest. It's a little bit of a debate among scholars whether people sought out confessors who assigned easier or tougher penances; the reality is presumably--both.
The wild efforts of people to avoid time in purgatory--indulgences much?--also point to the importance to them of confession, since that was the thing that would get them into purgatory instead of hell in the first place. (Indulgences do NOT save a person from damnation.)
Which brings us back to fama, surveillance, proof, and the possibility of blackmail.
Confession today conjures mental images of "the confessional," the closed little private box, hushed voices. The confessional is an early modern invention. While priests were required to keep the so-called seal of the confessional, the actual practice of it would be the penitent standing next to the priest with a long line of their neighbors standing right there--as popular literature attests, in a position to overhear all too easily.
And gossip, remember, easily became reality.
I can talk a little about some 18th and 19th century British instances of blackmail here, belatedly. The legal standard for proving sexual transgression between males, such as sodomy, could really vary across the 18th and 19th centuries, both with changing legal definitions and with the class background of the alleged participants. However, blackmail around sodomy allegations during this period could hinge on evidence as seemingly-circumstantial as the exchange of money or gifts, or simply being alone with one's accuser for a while in which sexual contact might have taken place. I should note that these things were grounds for suspicion because they frequently did accompany actual sodomy -- the exchange of money, goods, and favors for sexual services is an enduring phenomenon, and sex acts often take place in private, whether consensual and by agreement or nonconsensual and achieved through force or coercion. It's not necessarily easy to tell from the written legal records of this era whether an individual is defending themselves against blackmail concerning their substantially true actions or not, but the mechanics of blackmail might proceed in the same manner either way, with the accusation of damaging conduct serving to enforce secrecy as well as the urgency of paying up.
The prisoner then said, "I'll shoot your b[lasted] head off, if you don't give me all the money you've got." Witness eventually gave him one shilling, which he was induced to do through fear. Witness then turned to leave the room, when the prisoner seized hold of him, tore his shirt, coat, and smallclothes, and said he would give him in charge for an unnatural offence, if he did not give him more money. - Morning Advertiser, Tuesday 23 August 1842.
The formal legal penalties regarding even privately-conducted sexual activity between consenting adult men made contesting a sodomy accusation (whether formal or informal) potentially dicey. If sex between men had taken place and therefore the subsequent allegations of sodomy were true, the risk that such a thing might be proven in the eyes of the law and exposed beyond a doubt was a real and tangible threat. If no sex had taken place, the process of arraignment, investigation, and testimony was no more pleasant for it. Between the 1530s and the 1830s, the civil penalty for sodomy in England was death, and even lesser punishments like being sentenced to stand in the pillory were highly public and potentially lethal ordeals.
The charge of sodomy and its legal punishment were both sufficiently public that raising the specter of a sodomy allegation was often sufficient to achieve the end of extortion without a formal legal arraignment or the requirement of tangible material proof like incriminating letters or photographs. Corroborating documents like private letters and diaries did come into play in some cases of blackmail, but in the late 18th and early 19th centuries I wouldn't say that the legal standards regarding proof of sodomy were in actuality as low as the ease of blackmail might make them out to be across these centuries, particularly if the accused was a man of means and generally good reputation, but for individuals without financial resources or social capital to rely on, or whose reputations were especially significant, such accusations could be devastating with or without any tangible proof to back them up. To an extent this is because of the social and legal ramifications of a damaged reputation and an association with a despised crime, but it's also a function of how completely despised the figure of the sodomite was in the public imagination -- many social consequences for individuals believed to be sodomites took place entirely outside of the court system and before any evidence need be presented. In theory the act of buggery -- in this case, penetration of the penis into the anus, ideally concluding in ejaculation -- had to be proven in order for legal punishments to proceed, and ideally eyewitness testimony would support this conclusion, but the social consequences of such an accusation required no such codified standard of evidence.
In the 18th century, blackmail did not necessarily hinge on a pre-existing relationship with its target or require corroborating evidence in the form of letters or images -- it could occur between relative strangers. Once both blackmailer and target were in a position for sexual contact, or where sexual contact might have plausibly occurred, the blackmailer could demand payment. In many of the cases of this type that end up in contemporary print sources like newspapers and court records, the targets of blackmail protest that nothing improper had actually taken place, and that therefore complaints of buggery were unsubstantiated. Some targets for this ruse were likely oblivious and free from any sexual intention, while others had been targeted specifically because they were looking for sex and companionship from other men, even if short of buggery.
One such 18th-century case is rather unusual because of the personal background of the individual targeted for blackmail. In July of 1732, the London servant John Cooper testified that one Thomas Gordon had gotten him alone after a few drinks and robbed him of his clothes at knifepoint. Having achieved this aim, Gordon then threatened that if Cooper reported the crime he'd say that that Cooper had given him the clothes as payment for sex and was therefore guilty of sodomy.
I was got about 15 or 20 Yards off when the Prisoner came up to me, said it was a fine Morning, and ask'd me to take a Walk; I agreed, and we went into Chelsea Fields, and turning up to a private Place among some Trees, he clap'd his left Hand to the right Side of my Coat, and trip'd up my Heels, and holding a Knife to me, God damn ye says he, if ye offer to speak or stir I'll kill ye; give me your Ring. I gave it him, and he put it on his own Finger; then he made me pull of my Coat and Waistcoat, and Breeches; I begg'd that he would not kill me, nor leave me naked; No, says he, I'll only change wi'ye; came pull off your Shirt, and put on mine; so he stript, and drest himself in my Cloaths, and I put on his; there was 4 1/2 d. in my Breeches, and I found 3 ha' pence in his. He ask'd me where I liv'd, and I told him. I suppose, says he, you intend to charge me with a Robbery by and by, but if you do, I'll swear you're a Sodomite, and gave me the Cloaths to let you B[ugge]r me. - John Cooper, testifying at the trial of Thomas Gordon, July 1732
Afterward, Cooper had sought assistance first from strangers and then from a constable but was beaten bloody by several men who'd heard the allegations of sodomy from Gordon and who rebuked his victim as a molly. The threat of exposure wasn't enough to keep Cooper from seeking justice, however; he identified his assailant with the help of his neighbors and had the man brought before a justice of the peace. Though the case went to trial, Cooper did not receive the satisfaction of seeing Gordon hanged for his crime; Gordon was acquitted due to the positive character witnesses of several of his Drury Lane neighbors, but Cooper faced no apparent punishment either for cross-dressing or being a party to sodomy. So things didn't turn out great, but they didn't turn out terrible, considering it's quite likely that Cooper really was a molly -- a man who had sex with men, who perhaps crossdressed, and who participated in a subculture with other men who felt and behaved similarly. Cooper was reputed to run errands and messages for such mollies and several character witnesses in the trial testified that he was in the habit of attending masquerades in women's attire and dancing with "fine men".
If Cooper was wearing men's clothing at the time he was attacked, how did any of these men know he was a molly? Why would hearing that a man was a molly prompt total strangers to beat him until he reportedly spat blood? This should give at least a sense of how ingrained the impulse of violence toward accused sodomites was among the English public -- in addition to reporting and aiding in the detention of sodomites, the average English citizen of the Georgian era could do their part by assaulting and verbally abusing sodomites and their collaborators in the pillory or on their way to the gallows. Did the defendant suspect that Cooper was a molly before targeting him, or that he perhaps participated in sodomy on other occasions and with other men? Did Cooper's reputation for associating with wealthier mollies make him vulnerable to attack? Or was Cooper actually meeting with Gordon with the expectation of sex at the time of the robbery? Had sexual contact in fact taken place before Gordon forced Cooper to exchange clothes with him? None of these things are answered just from the written records of this case, but these events should suggest both the threatening power of a sodomy accusation -- not just to Cooper's reputation and social standing, but to Cooper's life and safety -- and the messy process of documenting what was then a criminal offense in an instance without third-party witnesses.