This is a case of the game playing to the cultural tropes of the time, generally speaking. Medieval millers had a reputation, earned or not, for dishonesty. The English phrase, "to put one's thumb on the scale" comes from the idea that miller's would cheat farmers when weighing their milled grain.

A good example of this would be Geoffrey Chaucer's Canterbury tales from 14th century England. One of Chaucer's pilgrims is himself a miller and is presented as a generally unpleasant and morally base character. The miller's tale is told in response to the knight's story of high courtly honour, and is all about cuckoldry, farts, misdirected kisses, and other slapstick comedy.

It's similar to the common fantasy trope that the criminal underground of a city will be found somewhere at the docks.

Edited: I nearly forgot mention as well that the Reeve in Canturbury tales directly accuses the miller of doing the kind of cheating I described above, and builds his own story to get back at the miller.