This was asked a few days ago in At the peak of slavery in the continental United States, what percentage of the population would have owned one or more slaves?. The top-rated answer, by /u/sowser, starts with a good discussion of what "owned" means, and that counting those who owned slaves greatly understates the number of people who controlled slaves.

A related question that you might find of interest is How financially privileged was slave ownership in America? Could a "middle class" American own slaves, or was this only reserved for the very elite? Do we have knowledge of what percentage of Americans owned one or more slaves?, where /u/sowser points out that most slaves were owned on large plantations. "In 1860, we estimate that about half of all slave owners in the Old South would have owned only a single slave; a similar percentage would have owned fewer than half a dozen across the entire South." The average (arithmetic mean) slave owner owned a little fewer than 10 slaves, "but the median slave was owned by someone who held twenty-two slaves overall, slightly higher if you exclude urban slave holding".

Usual disclaimer that I'm only mentioning these articles as previous writings here, not the be-all and end-all of answers; if you have more questions or want more expansion, you should consider asking.