I guess this question is more directed towards heavier slave owning regions, though if you know something about small Appalachian farmers, that’d be great too.
How many people owned land but not slaves? How many were employed to oversee slaves? Did white day-laborers have to compete with slave work gangs?
While it doesn't focus on the labor side of things too closely, you may find this to be of interest, which looks at the socio-political aspect, in relation to the Civil War, and does look at Appalachia, although not the whole focus.