It's hard to say for certain. Planters were definitely not the only people to own slaves. There's a well documented land owning middle and lower class who both owned slaves, so it's very possible that the ancestor in question did own at least a few slaves. If you're doing the research for this I can tell you an excellent way of finding out. Check out the probate records that were taken after someone died. In essence, when a person died all of their goods were documented for tax and debt reasons, so any slaves they had would definitely have been on that list. I did some work on South Carolina history that heavily relied on those probate records. Unfortunately, I don't think those documents are digitized, since there's a ton of them and they're generally organized in the archives by county. So, if you know what county this ancestor lived in and when he died there's a pretty good chance that you can find out.