The trope that only a small percentage of the southern population actually owned slaves is quite misleading.

It is often said "Only one in every 20" Southerners owned slaves, or "Only 5% of the Southern population were slave owners" but these are very deceptive statistics (and often associated with the “Lost Cause” narrative which the South long used to excuse the Civil War).

Children do not own slaves. Wives (generally) did not own slaves. Young people (until they inherited or were gifted slaves, or had enough resources to buy them themselves) did not own slaves. All these people, however, could easily be part of families which did own slaves.

A better statistic to measure how many Southerners were closely connected with slavery is to measure the % of households which owned slaves.

This was 26% in Virginia, and close to 50% in some states farther south. (49% in Mississippi, 46% in South Carolina). It was about 25% on average across all the slave states.

http://www.civilwarcauses.org/stat.htm

If you look at the percent of households owning slaves, you will see that slave ownership as a % of population gives a very misleading impression of how closely most Southerners were connected to slave ownership.

The majority of southerners were against slavery on religious grounds but there was a overwhelming cultural fear that by setting the black population free, they could rise up and overwhelm the white population.

An incredibly important ideal to note in this time period is the idea of paternalism. The white slave owners saw the absolishment as a "dig" as if society were saying that they had failed to take care of "their people." It's hard to see it that way sometimes but in a broad cultural sense that's what was happening. The general public appeasement of slavery came from a fearful cultural and economic place.