Slave labor wasn't inherently cheap, and slaves were large capital investments. Very few white southerners owned slaves. So the premise is flawed.

However, many farms, particularly in the Appalachian mountains and in the northern range of slavery, didn't have slaves. Slavery was heavily concentrated on the coastal plains, low country, and the Mississippi embayment, which today forms America's "black belt".

Whiskey manufacturing, especially in Kentucky, West Virginia, and Tennessee, was a common industry as it was easier to distill the corn grown west of the mountains and send it across as whiskey than it was to send it as grain.

Many built infrastructure as well, like the Santee Canal, and the railroads. They also were employed to oversee the plantations, as many plantation owners fancied living in Charleston or New Orleans.

That all said, many of the so called "Pine Barrens" we're desperately poor, and poor immigrant, largely Irish, slums existed in southern cities like Savannah. Many lived at a subsistence level of living standards.