They worked.

Cotton picking is the most iconic slave activity, but it was not the majority of the work a slave did. Tending to and processing the primary plantation crop was only 35% of the work a slave did. The next biggest was tending to livestock or growing their feed, about 25% of their time. Another 10% was other farming, such as corn or potatoes. The remaining 30% was construction, farm maintenance (ditch digging etc), cleaning, and especially artisanal production. (Note, these are averaged across a sample of plantations from Time on the Cross)

So roughly half of a slave's work was not seasonal. Animals still need to be cared for in winter. Spinning cloth, cleaning, and building structures can be done in most seasons. These non-seasonal tasks were done when the seasonal tasks were less pressing. So a slave would spend up to eighty hours a week harvesting cotton during harvest time and would, during the winter, spend more time spinning cloth, or building fences (or additions to the master's mansion...), etc.

In this sense, the slave experience was not especially different than a free farm laborer. They often used familiar tools, had familiar work patterns, and so on. But only in this sense: a free laborer was paid, could leave their employment at any time, and was less likely to experience violent coercion. If they did, it was considered legitimate to defend themselves or escape. There was also a cultural narrative that they 'deserved', if they worked hard, to eventually have their own land. The cultural narrative for free African Americans was precisely the opposite.

Anyway, to address the idea of vacations: The average slave worked 270 days a year, an average of about 50-60 hours a week, skewed with more work in harvest time and less in winter. This had led to some people claiming they had roughly three months 'off'. The truth is more likely that slaves were often let off on Sundays and had a few other days or the odd week off.