Point of clarity before going ahead: I'm using the term "heretic" here to describe any person of any denomination that faced judicial proceedings for not conforming to the confession of a particular area. A Catholic in England or a Protestant in Spain both fall in this category, as would an Anabaptist anywhere in Europe besides the Ottoman Empire.

Aside from the clergy, it's not as if Catholics, Lutherans or Reformed Protestants dressed entirely differently or had other external signs showing their confessional affiliation. Occasionally there would be some sort of dress regulation within a community, but these were generally enforced at the local level. So for example, a Calvinist consistory or town council in the Protestant Netherlands or Languedoc might have certain regulations about dress or behavior, but these would apply to any covert Catholics that happened to be within the community as well.

Finding crypto-heretics then took some work. The Inquisitions are generally the most well-known searchers of infidelity, but they generally didn't go door-to-door seeking signs of heresy. An Inquisition court (or consistory or town council in Protestant lands) more often than not had people reporting suspicious activity. If someone stopped attending mass, or started baptizing other adults, or began preaching in the pubic square (like Ignatius of Loyola did to garner the attention of the Inquisition), that was generally a good sign that you had a heretic in your midst. Snitching was certainly something that people did, and accounts in Inquisition court records often relied on reputation as much as an observed action in dealing with potential heretics. On the other hand though, by the late sixteenth and early seventeenth century, local parishioners were more interested in sussing out witches than covert Protestants or Catholics.

If surveillance failed, the main way to figure out someone's denomination was to ask them. Yes, they could lie, and many people did lie. Lutheran, Reformed, Anabaptist and Catholic moral theologians all wrote vehemently against the "demonic temptation of apostasy," but there were qualifications made, especially in the latter half of the sixteenth century, to support individual cases of what was called Nicodemism. Lying aside, simply asking was a far more effective way of investigating heresy than one might think. More often than not, heretical thinking was just the result of misguided notions. Inquisitions generally sought to find out where the suspect had fallen into error and correct them, with punishment only coming later. For those that understood full well that they were committing heresy, many of them were okay with facing the penalty. The Reformation saw the resurgence of martyrdom as a transconfessional discourse. People were proud, sometimes even delighted at the idea that they would be killed for the promotion of their faith. Showing steadfast loyalty to the gospel truth (however one defined it) in the face of certain death and torture was an admirable trait, and one that would be rewarded in the afterlife.

Generally (and I have used that word a lot in this because nothing is the same everywhere) the people that were persecuted for heresy were publishing, preaching or openly speaking against the dominant confession of their area (and often against the local or royal authority therein). There are cases of clandestine heretics, such as Jesuits in England, being "discovered" and rooted out, but generally heresy was a public outburst rather than private wrongthink.

N.B: This is not a universal opinion. Some historians would point to the many pogroms or outbreaks of mob violence against suspected heretic groups as evidence against what I have just said. I am merely talking about heresy as a crime being punished in a normally operating government system. "Suspicious" persons (whether due to race, nationality or confession) within a community are consistently treated violently throughout history.

Sources:

I think a book that gets at the heart of your question is Perez Zagorin's Ways of Lying, even if it is not directly about surveilling dogma. One that is a bit more on point for that would be John Tedeschi's The Inquisition in Early Modern Europe, but it is a bit of a more difficult read. Brad Gregory's Salvation at Stake is indispensable in talking about martyrdom in the Reformation. And of course, no reading list about heresy in early modern Europe would be complete without mentioning Carlo Ginzburg's The Cheese and the Worms.