There are several things at play here

The first one is racial solidarity. Many people within the white South placed a very high premium on some sense of unity among white southerners. This sense of white solidarity played a significant role as a source of social and cultural cohesion; even if one did not oneself own slaves, the hope existed that one might in the future own slaves. At the very least, a poor white southern man at could at least know that someone else, the slave, existed at a lower social status than he did. Certain scholars have applied the term “herrenvolk” (master race) democracy to characterize the white southern worldview: a democratic society in which only whites enjoyed equality. This is related to the mudsill thesis expounded by James Henry Hammond of South Carolina: a functioning society needs to have some bottom rung that does all of the unpleasant work, and for the American South, enslaved black people played that role. Instead of a working class solidarity, there is kind of racial solidarity.

There is a gender component to this as well. Stephanie McCurry makes the argument in “The Two Faces of Republicanism” that the defense of slavery in the white South was at least partially a defense of the authority of white men within their households. One thing you have to understand is that for many white southerners, the slave plantation was itself an extended household with the planter himself as the patriarch. Just as a patriarch expected authority over his wife and children, he expected to have authority over his slaves. The subjugation of slaves, in this respect, could be defended as an extension of the submission of wives and daughters. Bear in mind that most slaveowners did not own huge plantations, so this defense worked even for smallholders who owned only one or two slaves. Even white southern who did not themselves own slaves still expected to marry, so maintaining these kinds of hierarchical social relations were still important so long as they could have someone underneath them in the hierarchy. You can explain a lot of southern culture and politics as a defense of the agency and authority of married, property-owning men.

There are some other complicated cultural factors to account for as well. There was anti-slavery sentiment in the South, but it was consistently more conservative and more restrained than what you would find in the North from the 1830s or so on, and it emphasized different things. Northern abolitionism/antislavery sentiment, especially from the 1830s onwards, was closely connected to romanticism and evangelical Christianity. Southern anti-slavery efforts in the 19th century were still emphasizing what is called “colonization,” ending slavery gradually and then deporting freedpeople out of the United States (this is why Liberia was created). One of the more famous southern opponents of slavery was North Carolina’s Hinton Rown Helper, who famously argued against slavery on the grounds that it was harmful to the region’s white population by excessively empowering the region’s planter “slavocracy.” The thing is, by the time Helper published his book, it was already too controversial a position to hold in the South, and had to be published in New York. White southerners also commonly disparaged slave traders and slave overseers; they would defend the institution itself to the hilt, but did not want to be associated with certain aspects of it.

So there was some antislavery sentiment within the South, and while I wouldn’t say that the sentiments you’re referring to were absent, there were other complicated social and cultural factors that made its defense important even to those who did not directly participate in it. Please let me know if I can clarify anything.

Readings

Stephanie McCurry, “The Two Faces of Republicanism,” Journal of American History

Lacy K. Ford, The Origins of Southern Radicalism: The South Carolina Upcountry, 1800 – 1860

Kenneth Greenberg, Masters and Statesmen: The Political Culture of American Slavery

Bertam Wyatt-Brown, Southern Honor: Ethics and Behavior in the Old South

Steven Deyle, Carry Me Back: The Domestic Slave Trade in American Life

James Oakes, The Ruling Race: The History of American Slaveholders

J. William Harris, Plain Folk and Gentry in a Slave Society: White Liberty and Black Slavery in Augusta’s Hinterlands

J. Thornton Mills, Power and Politics in a Slave Society: Alabama, 1800 –1860