Some of this was related to rural Britain hat styles. See, for example, this site. I don't think this provides the entire answer: it was probably a complex of factors that helped establish this stereotype, but the exotic nature of the people on the Celtic fringe (and their traditional - perceived - association with mystic, magical arts), was probably a factor.
edit: I added the word "perceived" because I don't want to imply that the Celts had or have the market cornered on mystic, magical arts, merely that many perceive it that way.