Cotton may have been 'king', but it was hardly the only crop being grown in the antebellum South, just the biggest. Cotton dominated the Deep South, stretching from East Texas into the uplands of South Carolina, but the Mississippi river delta of Louisiana was sugar country; the upper South, especially Virginia, did a lot of business in tobacco; and the low country of the Carolinas and Georgia was the 'Rice Kingdom'. Not to say that there wasn't overlap - such as the cotton growers on the Carolina Sea Islands - but the main point is to say that cotton was hardly the only cash crop of the South. And of course, just as in cotton country, rich white men weren't working their own fields, but instead kept large numbers of people enslaved to toil away.
Looking specifically at the rice plantations, while, as you say, ones image of the Southern plantation has the masses of enslaved persons toiling away in the cotton field, the rice plantations were actually the largest slave labor camps for the most part. Looking at the 1860 census, fourteen plantations were listed as holding more than 500 people in slavery, and eight of them were South Carolina rice plantations, with a ninth a rice plantation in Georgia. The largest, the estate of Lt. Gov. J.J. Ward, held a total of 1131 people in slavery. When you start looking at family groups, the seven scions of Nathaniel Heyward enslaved just under 3,000 people between them, over a million (1860) dollars in pure human capital.
And even more than that, due to how South Carolina's census taking calculated the numbers - strictly on a parish by parish basis - three more rich white rice selling enslavers actually held more than 500 people in bondage, but spread over multiple estates in several parishes. And again looking more broadly at family groups, the 22 families which made up the rice planting 'aristocracy' of South Carolina combined for the exploitation of 20,000 people's unpaid labor. Much more common of course, were the smaller families who even combined owned less than 100 people between them. 140 families held somewhere between 35 and 99 enslaved persons on 150 low-country plantations, and in total, the 320 rice plantations of the South Carolina Low Country, owned by 250 families, were the labor camp for 48,000 enslaved black persons - this in a region which only numbered 40,100 white people, and that still excludes any families who held less than 35 slaves!
The low country region was quite dominated by this planter aristocracy, with the tidal swamp lands near the rivers, moving up from the coast to about 40 miles inland being considered exclusively their domain. 9/10 of these lands were owned by this group of elite planters, and any small time planter or yeoman, even with a few slaves, unable to afford getting anywhere near. And while rice had become well eclipsed by cotton by the mid-1800s, rice had been the king crop of the 1700s, in that period a small cadre of white enslavers building their fortunes off the work of their black bondsmen, who at that point made up 80 percent of the lowland's population.
Now of course, while the land might have been prime real estate for wealthy white families, who could build their beautiful mansion and elaborate gardens - a number of which remain open for idyllic tours with barely a hint or mention of the centuries of exploitation on which they were built - as you might suspect, it wasn't quite the same for the people who actually were doing the work. As was mentioned, these plantations were on tidal swamps, and while it was believed that those of African descent were more resistant to the malaria that their working space bred, this was hardly comforting when the disease took away your loved ones. Enslaved children especially were quite susceptible. At Gowrie, a Georgia plantation of Charles Manigault, for instance, the records show 6 infants alive in 1833 when he first 'founded' his new plantation on the Savannah River, and all of them dead by 1835, one of cholera and the others from malaria, or else some enteric disease contracted from living in a literal swamp.
Even adults were hardly immune to the ravages, although at least with less than a 100 percent mortality rate. Cholera, dysentery, malaria... in one particularly brutal span, 1/3 of the enslaved population of Gowrie died of disease between 1846 and 1849, and Manigault was only able to keep the plantation running - and at a healthy profit - by buying new human beings to replace those who he had already worked to death. Although instability of family life was one of many curses visited upon the enslaved by their white masters, the disease ridden conditions of the 'rice kingdom' was an especially awful addition to that burden.
So anyways, while I can't say much about the Louisiana Sugar plantations, and only slightly more about tobacco operations in Virginia, I hope this offers a small look use of enslaved labor in something other than cotton. To be sure, there are broad similarities, and it isn't that hard to change the image in your mind from an overseer driving slaves in a cotton field to that of a rice paddy, but the conditions found for slaves in the Low Country had their own certain kinds of horrors. And of course too, the Low Country bears note for being one of the most enslaved regions of the country, where the white masters were outnumbered by the persons who they exploited.
For more, see:
"Them Dark Days: Slavery in the American Rice Swamps" by William Dusinberre
"Masters of Small Worlds: Yeoman Households, Gender Relations. & the Political Culture of the Antebellum South Carolina Low Country" by Stephanie McCurry
As a follow up question, were slaves who worked at professions in cities (e.g. smithing) given more leeway as opposed to those on a plantation?
In this vein: how widespread was slave labor for major construction projects? We often hear in the U.S. that "slaves built D.C.," with special focus on the Capitol and the White House. Was slave labor a common feature of antebellum American construction?