Is there any specific bad smell you associate with them?

Well in an era of hard physical labour,long working hours, limited access to washing facilities and clothes that were worn for many days at a time then, yes, to our modern deodorant rich sensibility people in the past are likely to have had a stronger smell about them.

For example, Chekhov described ballerinas as smelling like horses, presumably from the stale sweat of dancing. Wolmar describes the distinctive smell of working men in relation to porters in goods yards. James Boswell was described as having a strong distinctive occasionally overpowering personal odour, with which he was still welcome in fine society.