Good question. As with everything in history, it depends where you are, when you are, the people, culture, who rivals were etc. but for the sake of clarity I will pick out Anglo-Saxon and Norse past-times.

It's actually not that big of a surprise but they loved games. Board games, dice games, competitive games/sports and riddles specifically. They were a good way to pass the time and often allowed the prowess of one to be shown. They took their gambling extremely seriously and there are a few cases that some people would even bet their freedom on a game. Gambling was a measure of ones skill and luck; the more luck you had, the more you were admired. Likewise, riddles allowed for humor as well as intelligence to be displayed. One of the most well known board games is a game called Hnefatafl, a game not entirely unlike chess where the king is targeted.

They certainly do a better job describing it than I would: http://tafl.cyningstan.org.uk/

They also adopted and adapted a number of Roman games including Duodecium Scripta which they renamed Kvatrutafl. Again this game is not unlike present day Backgammon where the objective is to move your pieces to home ground before your opponent.

Here is the Wiki for it: http://en.wikipedia.org/wiki/Ludus_duodecim_scriptorum

Another game they played was Chess (late medieval period) as well as knucklebones which are both still popular today in one form or another.

There are plenty more if you are interested but the last one I will touch on that I have found in my research is song/poem writing and story-telling. A talented musician or songwriter in their off-time may make some money on the side. A very talented minstrel may even find himself lucky enough to be hired by the local elite to play on a permanent basis. However, for the average peasant who lived hand-to-mouth (not necessarily destitute poor, but not wealthy) may have stuck to storytelling.