Wilma Dunaway's "Slavery in the American Mountain South" explores this topic at length in the context of the Appalachian region, which generally had a much lower percentage of it's population enslaved than coastal regions, and thus, the area's slave owners tended to own far fewer people.

TLDR of it is that it could be extremely isolating. While the enslaved on large plantations like Monticello and Mt. Vernon could count on essentially having their own community of other enslaved people on the plantation to support each other, have families, etc, the same is not true of an enslaved person in a household where they might be the only enslaved person there. Another big difference is that on plantations the enslaved usually have their own quarters which are often set up like a neighborhood (Monticello's Mulberry row being a great example) while small slaveholders were much more likely to keep their enslaved living somewhere in their house. While some people might think that sounds nice, the reality is that living in your enslavers house means living under their gaze 24/7 with zero space to yourself.

She also explores the trend of people renting out slaves for temporary use, and that institutions like churches sometimes made a tidy profit by acting as slave rental agencies. If you've ever heard the phrase "drive it like a rental" in the context of cars, imagine that logic applied to a person. Dunaway cites court cases in which renters are sued by the owners of these slaves for "damaging their property" during the course of these rental leases through abuse, inadequate food or clothing, etc.

Finally, she talks about the use of enslaved people in more industrial operations such as iron furnaces and mines.

You should read Narrative of the life of Frederick Douglass, an American slave.

While each individual slave owner was different and there is no one size fits all answer to this question, Douglass offers some of his personal experiences that can help shed light on this topic.

Both scenarios would have their own pros and cons.

One of the biggest downsides of a larger plantation would be that they have more slaves and thus there would be overseers (white men that would be in the fields literally cracking the whips and keeping them inline). These overseers had to look like they were effective to the masters in order to show they were providing value. Douglass depicts them as being especially brutal.

Douglass recounts of an overseer named Mr. Plummer:

I have known him to cut and slash a women’s head so horribly, that even master would be enraged at his cruelty, and would threaten to whip him if he did not mind himself. Master, however, was not a humane slave holder.

I believe that another reason why the overseers were especially brutal is due to that fact that they were lower on the social ladder. The fact that slaves were even lower gave them a chance to actually feel superior and would abuse this power over them.

Another downside of being on a large plantation was the fact that if you did something that warranted a punishment, you would be made an example out of and brutally punished.

Another downside with being on a merger plantation is that the master was very connected in society. Often the master would have extended family members and professional connections where slaves would be loaned to. Douglass was sent to the farm of a poor person who was known as a ruthless slave breaker. The masters would loan slaves that tried to escape to this man to punish them and get them to lose the will to escape.

If all else failed, a well connected master could also sell you to another master that they knew was exceptionally ruthless. Frederick Douglass notes in his narrative that while all slavery is evil, he was lucky that he was in Maryland instead of the Deep South. There were times where his master threatened to sell slaves to someone in Georgia or the deep south where the brutality was even more extreme.

One of the benefits of being on a larger plantation is that there were a multitude of jobs. You could potentially get a job that wasn’t as difficult as others ie in the house instead of the field.

Another benefit is that You also had the possibility of being sent to a family member that lived in a city. This happened to Douglass and that is where he was able to start to learn to read.

When it comes to the negatives of smaller farms, the same notion of poverty and the frustration of social status was often taken out on the few slaves the farmer had.

Smaller farms also didn’t have all of the same connections. This is a mixed bag. If the master was brutal, you were stuck. If the master was relatively better, you not as worried about being sent to a worse place.

Slaves were consider assets like livestock. Masters of large and small estates would rape their slaves. This gave the masters sexual pleasure, but also often produced more slaves. It is believed that Douglass was the son of his first master. This was common on all sizes of plantations. I won’t say that one size was more inclined to do this, but if you were a poor farmer with only a few slaves, the marginal benefit of getting one of your slaves to have a kid was much larger than if you already owned hundreds of slaves.

It was common place to have the older women raise the children while the younger women did more of the physical labor. One brutal aspect that Douglass points out is that his grandmother raised him as a young boy and when she was too old to be useful, she was sent in the woods to live alone in an old hut and die. Another common thing that often happened to slaves was that family members were separated from each other and sold to other slave owners. Slaves were encouraged to be married and produce children, but siblings would be separated, and children were taken from their parents to be raised by older slaves until they could start working or would fetch the best price at market. I don’t know if smaller plantations were this keen on separating families as the larger ones were.

All in all, life as a slave was terrible regardless of your location and who your owner was. Douglass mentions this in writings and speeches. That being said, being in a relatively northern slave state and living in a large city like Baltimore was leaps and bounds better than living in the Deep South and subject to that brutality and whatever idiosyncrasies your particular master displayed in how they treated and punished you.

Another thing that Douglass points out that nobody really thinks about is the interactions that slaves had with slaves owned by someone else. These slaves had nothing, so they would often brag or fight other slaves over who had the richest or more affluent master. So as desperate as it sounds, if you had a poor master, you had less to brag about to other slaves.