Poor people (though you're more likely thinking of peasants, who were not necessarily poor) worked through much of the day, either for their livelihood or tending to the needs of their homes, family, or church. Mending the house, washing clothes, cooking food, taking care of animals and pets, and cleaning up all had their places, and no one was exempt except for very small children.

However, they would also (sometimes depending on gender) take time to visit friends, participate in sewing circles, and/or take turns reading to each other, if they were literate (which might be a larger segment than you'd otherwise think). They might get together to tell stories, drink and carouse, play games or gamble, listen to the latest political or church news (sometimes the same thing), train for battle (occasional: think reserve training), or attend a faire. Sometimes, they would travel to other towns, especially if there was some sightseeing available such as a church relic, but also just to visit friends and family.

The number of things they had to do was far less than we have now (or even that might have been had 500 years ago), but it wasn't a boring, binary world of just work and sex.