I'm figuring that if slaves were the ones growing these crops, t hey would have basically unlimited access to cannabis. Surely, some of them must've snuck a bit back to their quarters for personal use. But this seems like the kind of thing that might have been wanted to be kept a secret, so maybe it's a stretch.
If you check the FAQ, you'll see that the subject of marijuana use has been dealt with many times. Just a few weeks ago, as it happens, there was a big discussion about pot in colonial America. The short answer is that no one in colonial America smoked weed. At all. The habit of smoking marijuana appears to have arrived from Mexico in the early 20th century.