Simply put, it is impossible to separate "Southern culture" from the ability to be an enslaver. The general perception that such a thing is even possible is evidence of how incredibly effective "Lost Cause" propagandists were, especially the white women who lead the United Daughters of the Confederacy.

Historian Karen L. Cox wrote one of the, if not the, definitive histories of the group, Dixie's Daughters: The United Daughters of the Confederacy and the Preservation of Confederate Culture and opens her book by explaining:

The term Confederate culture is used to describe those ideas and symbols that Lost Cause devotees associated with the former Confederacy. The images and beliefs are based on a hierarchy of race and class and often reflect the patrician outlook of Lost Cause leaders. Confederate culture to be sure, is based on the historical memory of its believers and is often racist. The Old South is idealized as a place where a benevolent planter class worked in harmony with its faithful and contented labor force. Within this culture, women remain wedded to their traditionally prescribed roles. Confederate soldiers are remembered as heroes in spite of military defeat, because they fought to defend states’ rights.

In other words, there is no white "Southern culture" from before the Civil War that is not part of Confederate culture. There were, of course, enclaves of free Black families who maintained their own culture. Many of these communities interacted with formerly - or actively - enslaved Black adults and children, most notably the the Gullah Geechee people. There was also the complex cultures and communities of the "Five Civilized Tribes" (Cherokee, Chickasaw, Choctaw, Muscogee, Seminole) whose ancestral roots are in the American southeast. When, though, people speak of a "Southern culture," it's generally in reference to what Dr. Cox described: an idealized place where the white planter class care for and lived in peace with their laborers. Or, as the song lyrics imply, a white man and his family could live simply on the land, even if they were poor, without bothering anyone.

There are multiple aspects of the human experience that can fall under the umbrella of culture (food, music, dress, social norms, etc. etc.) and we can see efforts to shape these aspects as part of some mythical ideal of "Southern culture" long before the Civil War. (Some more here from /u/partymoses about those efforts after the War and here for more from /u/Georgy_K_Zhukov about language and The Civil War.) One way we can get a sense of how that happened is to look at what cultural cues are actively passed down to children through formal and informal education. While there was a limited formal education system in the American south, there were a variety of structures for passing along cultural norms. I've written before about the popularity of juvenile literature and one thing I want to stress is the authors' explicit goal of describing an idealized South. In her stories, enslaved people were never harmed, but gently corrected if they got something wrong. They delighted in the smallest of gifts from their enslaver and took pride in caring for the land. White children might remark upon an enslaved person's lack of education or their inability to recognize the allusions to slavery in the hymns they sang, but were told not to comment on it. The gist of nearly every text was that white children were better than and entitled to be treated better than every Black person.

The audience for this literature wasn't just the children of enslavers, but also white parents and children who did not own slaves or may have lived far from plantations and only had passing experiences with Black children and adults (although it needs to be said that the literacy rates among most Southern white families was very low.) In effect, in order for a society to maintain certain cultural norms, every member of that society needs to play a role in endorsing or embodying those norms. This meant that it was critical that all white children (and adults) who interacted with Black people understood and helped maintain the social hierarchy where, no matter their class, a white child always had higher social standing than a Black adult, regardless of their class or status as a free or enslaved person. (There were some exceptions to this, especially as it related to child-minding on plantations where a Black enslaved woman responsible for caring for the enslavers' children might have the authority to reprimand or otherwise scold a white child but such dynamics were idiosyncratic and could be withdrawn at a moment's notice, based on the whim of the enslaver. Books such as They Were Her Property by Stephanie Jones-Rogers and Born in Bondage by Marie Jenkins Schwartz get into these relationships in more detail.) At a practical level, it meant that a white child was taught they could speak to a Black adult in anyway they wanted, they could ignore requests from a Black person, and most importantly for the purpose of your question, everything related to southern culture belong to them, not to the Black people who lived there.

The reason this detail is so critical is that it demonstrates how white Southern culture cannot be separated from the system of chattel slavery - race shaped every aspect of culture, even when it wasn't explicitly named as such.