I'm by no means an expert on this, but I am a very avid reader of 19th century literature (particularly Russian though).

From what I've read, I gather that people lounged together a lot (some times someone read outloud, some times some one read quietly while the women knitted or played music, other times they just conversed), they also went out for long walks, they played tabletop games, rural inhabitants went out hunting, city dwellers often went to social events and the theatre, and most importantly to understand this: it appears that work and leisure were not as sharply divided as they are now, so many people spent a large portion of their day doing work.

It depends very much on gender/age/class, but for common (i.e. middle/lower class), dancing, pool/billiards, some gambling, dog/rat fights, whoring (more for dudes, obvs :)), gaming, and music were popular.