This is a less than satisfying answer, but as no one else has commented yet I'll take a stab at it...

Per The Oxford Dictionary of Word Origins (2nd ed.), as part of their definition of "man", they also include that "a clergyman is a man of the cloth. The writer Jonathan Swift first used 'the cloth' to refer to the clergy in 1710. A clergyman's 'cloth' had meant his profession since the mid 17th century, and before that other occupations which stipulated a special dress code or uniform, notably the law and military, had also been referred to as a person's 'cloth'."