/u/Mddcat04 is absolutely right that just looking at the number of people who held legal title to a slave is drastically undercounting the white South's investment in the institution. The actual number of people with an investment of a kind is even bigger than that of slaveholding families. Beyond thm, one those who rented or took loans of slaves. These people don't own slaves legally, but are functionally slaveholders. Renting was a big deal particularly in the Upper South, but it happened often enough elsewhere that enslavers worried it led to slaves having too much freedom of movement and ability to set the terms of their own work.

And then there's the social and economic role that large planters often played in the lives of smaller farmers. Eugene Genovese has a good case study of how this worked. His example, "Joshua Venable, dirt farmer of Hinds County, Mississippi" owned no slaves himself but was cousin to one of the larger planters in the area, Jefferson Venable.

Now, of course, Josh resented his cousin-so much so that he continued to hope that he would someday own even more slaves himself and maybe even reach the pinnacle of success-some day he might be able to make Cousin Jeff a low-interest loan to cover his famous gambling debts, not to mention those debts for somewhat unclear expenditures in New Orleans. But, how far could he carry his resentment toward Cousin Jeff? Everyone, including Josh, knew that his cousin [...] always had a helping hand for anyone in the neighborhood, black or white. Josh raised some extra corn and a few hogs. What was he supposed to do, hand-carry them to Cincinnati? Wait to sell them to unreliable drovers, who specialized in hard bargains? Cousin Jeff was already ready to pay a fair price even though he could just as easily have increased the orders through his factors and not bothered with such local trivia.

Other services provided included the use of the Big House cotton gin for a small cost ("9 or 10 per cent) and bundling small cotton crops into his own to get a better price. Through the economics is a lot of this, there are also more neighborly social things going on. Joshua could expect an invitation to the big outdoor barbecues, where as a relation he would get a grand welcome. And Jefferson

was quick to send food supplies to help someone down-and-out. And everyone saw how he sent a few of his hands to help a sick neighbor get in his small crop when everything hung in the balance. If it were not for Ole Jeff and a few others like him, how could many of the poorer farmers make it?

A small farmer could also hope one of his sons might get hired as an overseer, which was one of the best ways to get yourself established as a planter before long. That would mean managing slaves, functioning as a slaveholder, which would then culminate in becoming one. If you got along on that road through some good fortune, but needed to buy a person before you could really put them to work then the Big House could rent that slave. Or if you had a temporary labor shortage and couldn't get a slave, you could rent yourself. For that matter, if you're new to the area and need a house Jefferson or someone like him might just dispatch a group of slaves to raise it for you.

Even if one didn't own slaves oneself, or come from a slaveholding household, there's a lot of personal contact with a buy-in for slavery in all that. Note Josh's aspirations involve getting slaves of his own. That's very common, since white labor was frequently scarce (everybody wanted his own farm and taking orders from a white man had a big emotional punch in a society where that was the defining trait of slaves), transitory (if you couldn't make it in this state, you might try the next one over in a year), and if you get right down to it, there are just things you could not get away with doing to white laborers to increase their productivity. To be a man, to be independent, a success, to be someone in the Antebellum South comes back around to slaves. You demonstrated your ability in governing them, in buying them at the market, in using them to make yourself rich. Freedom meant, sometimes very literally, mastery. They were, as Walter Johnson puts it, "made of slaves."

But even if you couldn't get there, and it became harder as slave prices climbed in the 1850s, then you could show your mastery in less direct ways. You might serve on the slave patrol, tasked with finding and capturing those slaves who dared steal themselves. You might participate in ad hoc operations against real or suspected slave uprisings. You could behave haughtily and disdainfully toward slaves on the street, treating them in ways that no white man would stand for.

And you could walk down to the town square, or go by the Big House, and see the slaves -including kids barely into puberty- stripped and whipped until they screamed and begged for relief, blood streaming. You might even see brine applied to the wounds. You could see that and know that it could never happen to you because you were white. Every stroke of the lash and scream of pain is another affirmation of your freedom from the same. These aren't things you necessarily get paid for in tangible commodities like cash, but they are ways that one can participate in and realize social and emotional capital in a slave society.

The planter line was always that slavery, by making a visually distinct race of "inferiors" also rendered all white men equal. The poorer whites knew full well that they were not equal, but they could also see how their shared whiteness made them akin to the planter and separated both from the slave. They had real investments in the existing order, even if their personal circumstances were less than exalted. They were not slaveholders in the strictest sense, but in innumerable ways great and small they were at least a peripheral part (and functionally, a far more central one) of the slaveholding class.

It's probably impossible to get a count of all of these and separate out those who were active participants and consumers of the South's master race democracy. There are certainly areas somewhat insulated from it and there are rare whites who resisted the system (many of whom found it wise to depart the South), but if you get right down to it slavery wasn't just something the direct slaveholders did. It was the organizing principle of society itself.

Sources:

The Genovese quotes, and much of the substance of this, are from Genovese, Eugene D. "Yeomen Farmers in a Slaveholders' Democracy." Agricultural History 49.2 (1975): 331-42. Web (Available through JSTOR's free reading program, which also kindly generated this citation.)

Walter Johnson's Soul By Soul and William W. Freehling's Road to Disunion are also useful for their understanding of the white Southern mindset.

That is an often quoted, but quite problematic stat. That number (generally 4-6%) is the percentage of individuals that owned slaves (here's the 1860 census results). What this fails to take into account is the number of slaveholding families. For example if you were a member of a large Southern family that owned slaves, only one member (presumably a father) would be counted as owning slaves. When you look at the numbers in the census data it breaks out number of slave owning families. This results in a much larger percentage of the South being involved in slavery, around 30%.

It's easy to see how people would make this mistake, where it becomes concerning is how it is often used. Personally I've seen this that used to justify the revisionist "Civil War wasn't about slavery" argument, usually presented as if only a tiny percent of the population owned slaves, they would have no reason to fight for the system.

• 1860 census results

What time period is this referring to?