So, this really depends on what you mean by the "average" southerner. The fairly accepted way to judge whether or not someone is a planter is by the number of slaves that they owned. We say that anyone who owned more than twenty slaves was a planter, because it seems that 20 slaves is roughly the point where the owner doesn't have to actually work on the farm. Of course, this is a rough measure since the life of someone with 19 slaves is not that different from the life of someone with 21 slaves. Either way, the number of planters is very small.

Now, to answer your question a bit more thoroughly, most "middle class" land owners in the south did own slaves. We call these land owners yeomen. The vast majority of them only owned a few slaves, generally less than eight or ten, which was enough to live comfortably and help get the crop in, but not enough to allow the owners to sit around. For them, slavery was still definitely helpful.

Further beneath them were non-slave holding landowners and non-land owners. These people owned no slaves, and therefore might be said to have no real interest in slavery. This is broadly true, but we do know that some planters lent their slaves to these poorer folk for free, thereby securing the planters' favor with the lower classes. Furthermore, there was a flourishing black market among the lower classes. It's a mistake to think that slaves never had any money at all. They frequently did and they either bought or sold things, or both through the black market. There's also a psychological aspect to this as well. Even the poorest white man could look at the slaves and think, "well, at least I'm better than them." This might not seem like much, but it was important to them.

So, to get to your minor questions. There are some important differences in the kind of crops that large scale and small scale farmers grew. Across most of the South, planters grew cotton. In a few select places, like Louisiana, they grew sugar or some other cash crop. Some yeomen also grew cotton, but the amount of cotton they grew is proportional to the number of slaves that they owned. Basically, the more slaves one owned, the more cotton one could grow. This is because growing cotton was a pretty risky business for a small farmer. You needed to take a loan to feed your family for an entire year in the hopes that your cotton crop would sell for enough to pay off that loan and make a profit. Most small farmers chose not to make that choice. They primarily grew subsistence crops, with a bit of cotton on the side, maybe. Now, the primary buyer for these subsistence crops was the local planter, who was growing cotton, but still needed to feed himself and his slaves. So, the crops that the majority of lower classes grew actually worked in concert with the big planters, not against them.

Another important aspect of all this is how interconnected all these factors were. For example, the planters owned most of the slaves, but they needed the cooperation of the rest of the citizens to help keep them from running away. They needed them to fill the ranks of slave patrols and keep an eye out for slaves sneaking around. To this end, there was a lot of favor spread through the community. Slaves were lent out, cotton 'gins were given out for free, planters held BBQs and other events to help secure loyalty.

In conclusion, it's very unclear whether slavery benefited the majority of society. Some historians have made the claim that slavery is at odds with capitalism and would have faded out as wage labor became more prevalent. I tend to disagree with this theory. There's no reason to think that slavery could not have adapted fine to a modern society. In fact, there are some examples of slaves working in factories and on railroads in the South. We can see the beginnings of change in the way slavery operated when the Civil War began and changed everything.

There's a ton of sources on this, so I'll just name a few of the big ones. Berton, In My Father's House are Many Mansions, Genovese, Roll Jordan, Roll, Bolton, Poor Whites of the Antebellum South, Barnes, et al, The Old South's Modern Worlds, Cecil-Fronsman, Common Whites, Forret, Race Relations at the Margins, Grivno, Gleanings of Freedom, Hadden, Slave Patrols, Harris, Plain Folk and Gentry, McCurry, Masters of Small Worlds, Wyatt-Brown, Southern Honor.