I would say it was the major cause of social tension in Southern society. Yeoman farmers in the south found it hard to compete with slaveholders both economically and politically, a good text on this topic is Masters of Small Worlds by Stephanie McCurry. Antislavery southerners found that they could not voice their opinions openly within their society and people suspected of antislavery attitudes were subjected to near lynchings (I'm familiar with one incident where white Ohioans suspected of abolitionist sentiment were tarred and feathered and forced out of The Kanawha Salines, a salt mining community I study, located near Charleston, WV). One book that documents all of the similar incidents in the south is American Mobbing by David Grimstead. Other abolitionist southerners, like Kentuckian Cassius Clay (a cousin of Henry Clay) had to physically defend themselves from mobs like the one that killed abolitionist newspaper editor Owen Lovejoy in Alton, Illinois.

And that's just a small example of how white southerners were divided by slavery. Enslaved people and free blacks were viewed as a constant source of social strife. Uprisings like Nat Turner's Rebellion, The Stono Rebellion, and the Denmark Vesey revolt left slaveholders paranoid about the threat of a slave revolt/uprising and forced them to create regular slave patrols, constantly harass free African Americans, driving many to migrate further north.