Most popular film and television dealing with Southern slavery (Gone with the Wind, Django Unchained, 12 Years a Slave, etc) deal with large plantations which would have had many slaves in industrial roles. Yet I'm sure there were a significant number of households that only had a few (1-4) slaves for housework, whom every member of the household would know personally. How common was this arrangement?
Contrary to the popular perception of south's slave society, the kind of slave-holding you describe was by far the norm. People tend to forget that slaves weren't just a source of cheap labour - they were also a capital asset, and whilst plantation slavery could be extremely profitable, it also required a significant capital investment before any return. Just like financial assets in our modern economy, this capital was disproportionately concentrated in the hands of the wealthiest members of society. By the eve of the Civil War, the top 2 - 3% of slave owners accounted for around one-quarter of all slave ownership in the south.
Something to appreciate though is that the pattern of slave-holding varies considerably from region to region and time to time; dramatisation and popular depictions of slavery tend to portray Upper South slave-holding as being a more benevolent kind of plantation system, but this was not the case. The average Deep South slave-holder owned about twice as many slaves as his counterpart in the Upper South. Whilst plantations existed throughout the region, they were overwhelmingly concentrated in the Deep South, where the commercial emphasis was on growing intensive crops like cotton. There, the wealthiest one-tenth of planters accounted for nearly three-quarters of all slave ownership by 1860. In contrast, in a state like Missouri, 90% of the state's slave-owning class had less than ten slaves; the average would have been around 5 per owner.
In terms of the very small, intimate kind of slave-holding you describe, we can reasonably estimate across the south that around half of all slave owners had fewer than five slaves to their name; most of these very smallholders would have been concentrated in the Upper South. Many of these owners would have still used their slaves for commercial farming but on a substantially smaller scale, working much less land with tighter profit margins. Throughout the history of the south though, the pattern was one of constant growth of the plantation economy - even in the Upper South.