Johnathan Rose's 'The Intellectual Life of the British Working Classes' goes into this in detail.

A couple of interesting things I remember from it: manual labourers (he gives blacksmiths as an example) would often hire boys to read them Shakespeare whilst they work. Another one, if I'm not mistaken, is that miners (depending on the working hours, country, date in time etc) would work relatively short hours due to the physical difficulty of the work and so have extensive hobbies which would include such things as you talk about.

I may have heard that second point from a different source though so I may be mistaken.