There's always more that can be said, but I have a past answer that deals with this:

"[J. F.] Kennedy despised how he looked in hats and usually refused to wear them - despite howls of protest from habderdashers whose hat sales took a precipitous downturn in the 1960s" (Barbara Perry). What factors led to the downturn in menswear hat sales in the 1960s?