I can't imagine generations from the 1800's (or before) being able to leave their farm, or having the money to do so, for the soul purpose of leisure time.
When did they become popular?
The latest episode of Backstory answers this question, especially the second segment: http://backstoryradio.org/shows/are-we-there-yet
Tl; dl - it happened when train travel became affordable to the middle class