I can't imagine that slaves, serfs and peasants had days off.

Well, regardless of whether you can imagine it, it did happen. This varied, of course, from one culture to another.

Here are a couple of previous AskHistorians discussions on the issue of slaves getting days off:

• July 20, 2014 post

• October 3, 2014 post.