My answer will cover years after the Revolutionary War, but one answer to your question would be that slave-owning was a privilege held by the upper class for the most part. For instance, by 1860, only 25% of whites families in the South owned slaves. Only ~1 to 2% of those slaveholders could be classified as a “planter”, a slaveowner who owns 10 or more slaves and 150+ acres. At this point in America, and especially the period you are referencing in your question, the bulk majority of slaveowners were British or of British decent. Those who would be classified as a planter would almost certainly be British, as acquiring that many slaves/acres of land would not easily be done in one generation. While there were instances of women, freed blacks, and native Americans owning slaves, the bulk majority of slaves would have been owned by white Protestant men of British decent.