From an earlier answer on tax fraud and preventing it in 9th-10th century Byzantium, with some new information about England for comparison:

Distribution of land ownership and use in the rural Byzantine Empire was definitely not even. By the eleventh century, there were some very powerful large-scale landholders (including monasteries, although at least in urban areas, sometimes even a monastery was controlled by an aristocrat), a few small landowners hanging in there, and then most of the population would have been paroikoi or dependent peasants: tenant farmers.

For the powerful landowners, including monasteries, tax collection was mostly a game of tax evasion. Tax collectors remained a popular bogeyman of Byzantine literature and art as they had been in Roman times, and this hostility was mirrored by practice. Owners might work to avoid periodic visits by imperial tax assessors to revaluate their land at a higher tax amount, pay their taxes using a debased currency, or send someone chasing after a tax collector to steal back the money. (Anna Komnene tells a similar story in her Alexiad.)

Most Byzantine people, however, were peasants who were responsible for paying their taxes (a) collectively and (b) often indirectly. The tenant farmers paid most of their tax burden to the owner of their land. In a case like that, they would presumably be fairly familiar with the people involved. Otherwise, taxes were levied collectively on a village. This had the advantage, for the peasants, of helping out if one family had a rough year. It had the disadvantage of--one of the reasons for this system--that local land abandoned or unfarmed was still part of the total levy for an area. So people had to cover for each other. But tax evasion was at least sometimes practiced by peasants--or at least, financial documents complain about peasants who simply disappear when they know a tax collector is in the area!

As for identifying the tax collectors themselves: at least some tax officials, most prominently the kommerkiarioi (who, as their name indicates, were involved with regulating commerce), had official imperial seals. This Google Books preview shows a couple of them, although they are both ninth century. Seals were pretty serious business in medieval Europe. The collective payment and indirect payment methods certainly suggest that at least some of the people involved would have been able to recognize the seal for what it was (medieval peasants are, as a rule, stereotyped in contemporary sources as stupid, but at least from later legal sources we can see a good deal of ability to negotiate successfully the demands of a more literate and educated "elite" culture).

Later medieval English manors evolved their own effective system of preventing impostor tax collectors: have the peasants elect one of their own to the job. Out of all the options for compulsory extra service, this wasn't the worst one in the world: it usually gave the collector a chance to skim some off the top for themselves. Well, it wasn't the worst until the Peasants' Revolt of 1381, when rent-collectors might find themselves lumped in with the targets of rebel anger and weaponry.