The poorer farmers worked the land in smaller amounts of those same crops or in necessary food crops to support themselves or sell at market. However, a large portion of the population would have been working in the trades to support the residents (rich and poor) in everyday needs. Be it a blacksmith to shoe the horses and fix locks on doors, the carpenter to build the structures, a tailor to clothe everyone (I Clothe All is literally their motto), etc. I'll use Williamsburg, VA as an example because it is so well studied and recorded. About 25% of all men over 16 were in trades. Maybe 2% were the larger plantation owners. The Fairfax Co. site has a really good rundown of sources about trades during this time. The rest fall into the smaller farm category. I can't personally speak much about the ability to grow such crops with a smaller number of people (keep in mind it wasn't just big planters that had slaves), but the infrastructure for small farms to sell the crops was certainly set up. Most often they would work through warehouses that would purchase the tobacco or other crops directly from the farmers to then ship them to England, acting as a sort of representative for those that could not afford their own private go-between. This is where the famous tobacco notes come in, being used in lieu of cash to pay the farmers. There were also regulations and inspections set in place by things like Gooch's Law. Virginia actually has an entire article on tobacco on their encyclopedia. There is some about the acreage workable per man in that article as well.