Occasionally I see people writing that only a small part (5% or so) of the white population in the southern United States owned slaves before the civil war. It is usually used as a way of claiming that slavery wasn't really very widespread, or even not a big issue.

I understand that owning slaves would be something that only rich people could do, but the figure of 5% seems like it's hiding something.

For every slave owner, there ought to be a wife, kids, employees that worked in overseeing the slaves, skilled workers whose jobs depended on the kind of big farms that were only possible when using slave labour, etc. Even if each slave owner only had four or five other people in their families and as employees who oversaw slavery, we are talking about 25 to 30% of the white or non-slave population.

My question is, how large part of the white (or non-slave, I don't know what is more correct) population in the slave states before the civil war got their livelihood from slavery, either as owners, dependents of owners or from jobs that depended on slavery?