I think you might be interested in this past answer by /u/sowser:

At the peak of slavery in the continental United States, what percentage of the population would have owned one or more slaves?

Then, as a follow-up to that, you might also be interested in this one by /u/Georgy_K_Zhukov:

Why were so many willing to fight for and defend the institution of slavery, even when they owned no slaves?

Follow up: Also more specifically, the southern Confederate states’ populations