This association always seemed really arbitrary to me, but it's such a pervasive relationship that I don't think most people even question it.
While I do not know the answer to this in enough detail to answer it here, I hope it's allowable to point you towards a source: Gayle Rubin's 1994 dissertation "The Valley of the Kings: Leathermen in San Francisco, 1960–1990". Probably not available in electronic form, unfortunately.