Reading about the region, it always perplexed me as to how despite pretty much only growing sugar they managed to support a large number of slaves and owners.
They could not grow all the food they needed! That is absolutely correct. The large sugar plantations of the Caribbean necessitated "bread box" colonies that could grow the food the sugar plantations needed.
The most famous of these is the present state of South Carolina. The rice plantations of the Carolina colony were all sending their produce down to the British Caribbean colonies, most frequently on ships built in New England (and carrying excess produce from there as well). Not so coincidentally, the rice plantations were the wealthiest enterprises in the thirteen colonies. Also, not so coincidentally South Carolina was the colony least willing to rebel, since they were the most tightly tied to the Caribbean colonies.
For the Spanish colonies, Mexico provided the food for the other industries in the form of wheat and cattle. While slavery was very much a part of Mexican history, the colony did not rely on it as much as the sugar or silver producing cColonies.
You asked about the French which brings up a fascinating "what if" scenario. In the late 1700s the French possessed the single most profitable sugar island in Saint Domingue, but they lacked a steady bread basket to feed it. Napoleon had plans to create plantations in the Louisiana territory to feed the slaves there, but the Haitian Revolution put a serious crimp in that plan. It was only after the Haitian Revolution that Napoleon agreed to sell Louisiana to the US since they no longer needed the potential food for Saint Domingue.
Which is not to say that the sugar plantations were intirely reliant on outside sources of food. First of all, most enslaved laborers had their own small plots of land on which they were supposed to grow food. However, the brutal work of cultivating sugar often limited the produce from those plots. Enslaved laborers working 20 hours a day processing sugar did not have any extra time to grow their own food.
But despite their image as monoculture, every sugar producing island had secondary and tertiary industries to provide food, oftentimes in the form of cattle or pork for livestock and corn, wheat, or rice for agriculture. The now famous "boucan" kept people fed in liminal cases. Hogs thrived in the Caribbean and could be hunted fairly easily. Cattle ranching remained a viable option on islands like Jamaica and ocassionally offered enslaved workers a chance at a freer and more comfortable life than the sugar plantations. More regularly, a "second class" of wealth developed on the islands. Planters, farmers, and even slaves whose economic success depended on providing food for the large sugar plantations.
All told, you are entirely right that sugar plantations could not provide their own food. But sugar as a commodity was only valuable in the context of a well developed trade web that made it easy for enslavers to get access to surplus food, either locally or abroad.
I will add a works cited when I can, but I am travelling at the moment.