Without listening to the podcast... the biggest danger, as someone working in the sugar fields (enslaved or as an overseer) was the presence of disease. Yellow fever and later malaria took huge numbers of people in the French colony of San Domingue. Deforestation in order to expand crop land, the subsequent erosion, and fly-by-night irrigation systems, created lots of disturbed soil- lots of places where water collected and stagnated for mosquitos to grow and carry disease. Further, diets were poor in the colony, especially among the enslaved laborers, so their immune systems were often compromised after making the middle passage and then subsisting of minimal nutrition. There is a terrible feedback loop between the 'seasoning period' or 'death's dividend'- the first three to five years, wherein most enslaved laborers died, and the economics of whether it was worth feeding them more hoping less would die, or if minimal rations were better as so many were doomed to die no matter what. So food was often limited- slaves could set up their own gardens in many cases- but that required them to survive long enough to do so. Yet, Europeans, who often had access to better food- though nothing spectacular as the business of the colony was sugar and coffee, not edibles, still died in pretty great numbers. Consider in neighboring Jamaica, while more enslaved Africans died numerically, in proportion to their total population in the colony, a European was more likely to die while in the colony. So that there was a fairly constant churn of enslaved Africans and also European managers. For many of the French and British marines, being stationed in the Caribbean to watch over the sugar colonies was akin to a death sentence because of the prevalence of disease and contaminated water-- infrastructure was simply more often than not left to the bare minimum. These were places to make money, very few colonists ever went to the colony planning to settle and stay there.

So the expansion of sugar production helped create the sort of environment that favored the multiplication of mosquitos-- many species of which made the journey from Africa to the Americas thanks to the expanding Atlantic slave trade, and who served as primary vectors for the disease. Sugar refining further upset ecological balance on the island, by requiring huge amounts of timber for firing the boilers to purify the sugar- producing molasses as a by product and then sugar crystals to ship back to Europe.

And of course I haven't said anything about the sheer brutality of coercing a labor force into sugar production-- especially grave when that labor force is seen as entirely expendable and replaceable.