It was a constant fear, both on a grand Haitian scale of a kill-all-the-whites crusade and on the individual level of slaves murdering individual enslavers. A large part of what otherwise seems like wildly disproportionate fear of challenges to slavery arises from the specter of a revolt, abetted by occasional testimony tortured out of suspected revolt conspirators that they had encouragement from learning that they had white allies and/or the Haitian example.

For white allies, some enslaved people accused of taking part in Denmark Vesey's planned revolt (or alleged planned revolt, there's some debate on it) name dropped Rufus King and other participants on the restrictionist (get slavery out) side of the Missouri controversy. Whether they were informed and gave up that information or had it basically fed to them by their torturers is a significant question but probably one we can't answer. Either way, the whites believed it was true and the revolt conspiracy was used as grounds to crack down on even the limited forms of white dissent from slavery that existed in the South Carolina of the 1820s.

Haiti got very bloody and the homegrown uprisings could on a smaller scale, plus even the conspiracies or "conspiracies" averted come replete with word of the gory intentions of the slaves. Look at that in the context of a profoundly intimate institution like slavery. While we should rightly picture most slaves doing work in fields and housed in separate and inferior quarters from their enslavers, most anybody who owns a person and exploits that person for their pleasure and profit also has slaves who live under the same roof and might sleep in the bedroom with them, or just outside the door to always be on hand. Slaves cook their meals and clean their clothes. They look after the children. For anybody with a decent-sized operation, they greatly outnumber the whites. (This can extend to whole counties or, in Mississippi and South Carolina, whole states.) Enslaved people are everywhere, all the time, deeply interwoven into the enslaver's life.

So if they resent all the horrific abuse being doled out, as everybody knows that they do, then they have almost infinite opportunities to get their revenge. Any slave who gives a dirty look might be planning something. Any time a white person gets sick, you have to wonder if it was an ordinary illness or poison. Every fire might be an arson. And where do they go off to at night when you can't see? What do they do when they meet away from the enslaver's watchful eye? They might just be hanging out...but what if they're not? The constant slow trickle of broken tools, "misunderstood" instructions, slow work, and all the other ways that slaves push back against the system can all seem of a piece.

To be sure, there's also a lot of talk about how an enslaver knows his people and they would never revolt, but it's belied by the constant fear of others and the draconian discipline meted out day to day. Samuel Cartwright is a great case study in both attitudes. He argued that black people, who he believed to be a separate species, were born for slavery. Their nature made them submissive knee-benders and they could not prosper otherwise. When kept in that state, they would be happy and healthy. In other words: slaves are fine with slavery.

Cartwright knew that this wasn't quite so. He could see all the everyday resistance that slaves offered firsthand. To square that with the rest, he insisted that Africans had a very different nervous system (smaller brains, more peripheral nerves...which he credited for their unique form of dancing) but mainly leaned on mental illness. Running away was Drapetomania and slow, slothful work was Dysaesthesia aethiopica. The slaves weren't unhappy; they were just crazy. A doctor (and Cartwright had a pretty top-notch medical education for his time) could cure them, but one probably didn't need the cost of a consultation. Rather, one could just order one's overseer to whip the devil out of the patient. (Or do it themselves, of course. Some enslavers liked that kind of thing.) Stern, swift correction would straighten the enslaved person right up by invoking their inborn need to submit to the superior white man.

Which is what they were going to do anyway. The quandary facing Cartwright is the same one that had faced enslavers for generations and his remedy was basically the usual. He's just one of many giving the imprimatur of science to extant social conventions. His medical rationalizations run in the same groove as the folk wisdom that the enslaved were by nature childlike and easily led. Since they would believe anything, one had to take a hard line lest they be tempted into something frightful like wanting freedom and being inclined to fight for it.

Sources

This is one of those things so omnipresent it's hard to settle on individual works. Cartwright is online in a few places, but most of them are decidedly unsavory. Here he is on Google Books. The article starts a ways before this, but there's quite a lot of intellectual throat-clearing and discussion of taxonomy that's less relevant than his signature diagnoses. I've also got it in a volume of DeBow's Review somewhere, if you're interested and that link isn't working.

For the Missouri Controversy, the modern treatment is Robert Pierce Forbes' The Missouri Compromise and its Aftermath.

William W. Freehling writes about Vesey as background for the Nullification Crisis (a connection he's less sure of these days) in Prelude to Civil War and takes up the general theme of slave revolt in his Road to Disunion (2 vols), though there he's more concerned with white dissenters than black resistance. The peculiar vulnerability of slavery to dissent and challenge and how it played out in national politics is also a focus of Cooper's Liberty and Slavery.