Short answer: they did plenty, though not necessarily on the physical side of things. Think of a Southern pre-Civil War plantation within the framework of a modern company, with workers and management staff as sharply divided in terms of responsibilities/duties. Just because the workers aren't paid doesn't mean that there isn't higher level stuff to do, just as in a modern company where the labor force is paid. Slaveowners oversaw the operation of their businesses just like managers do in modern times: i.e., negotiating prices for the commodities they produced, organizing delivery of that same commodity, paying state and federal taxes, directing the efforts of middle management (overseeers), negotiating the purchase of new slaves, seeing to infrastructure upkeep and expansion.

I don't mean to imply that being a slaveholder was an admirable or respectable thing, but if looking at things from a purely economic point of view (evaluating their function within the socio-economic system) they certainly had a lot to do. They were responsible for the ongoing operations of their plantations, farms, etc, and were thus tasked with the non-labor responsibilities (examples mentioned above).

[Sources: 'Black Rice' by Judith Carney; 'Many Thousands Gone' by Ira Berlin]