I think people tend to gloss over the transition from indentured Irish laborers to african slaves in the southern US and the Caribbean. Historians generally say that it is because africans were stronger and more able to do manual labor in the heat and sun. I think there may be some truth to that, but was that really the main reason? Irish were catholic, and England was flip flopping around in catholic vs protestant civil wars at about this time, could victimizing catholics have gone out of favor? And is this where our modern racism comes from, that somehow africans are stronger but dumber than europeans?
The transition from white indentured servitude to black chattel slavery has had a ton written about it thus far. I know it isn't the point you were making, but because you mention the Irish I'll link /u/Sowser 's great post on the Irish slavery myth regardless, here.
A key component is that indentured servitude was premised on short contracts and land grants. People in England would sell themselves as workers for X number of years. In exchange, they'd receive a land grant once they had finished the agreed upon period of labor. Maybe it wasn't a huge property and maybe not on the best land, but if you had not a penny to your name it wasn't the worst strategy for social mobility among many poorer people in early Modern England. This system had several key implications.
First, as the workers were on loan, employers had every incentive to work them to the bone while they were in their service. They had legal protections defending them against particularly terrible forms of physical abuse, but if it was between a poor, illiterate worker with few connections and a planter who is friends with the judge or governor, these defenses could mean little in practice.
Second, servants were making investments in their futures. Indentured servants had to be attracted with something, in this case land, in order to convince them to sacrifice so many of their best years under potentially terrible conditions.
Third, you had to have enough land to continue issuing land grants. In the case of a small island like Barbados, you could run out of additional land to grant and yet still need a constant influx of new workers. Workers which, going back to point two, now have little incentive to come. (for the above section, see Eric Williams: Capitalism and Slavery as well as The Cambridge Economic History of the United States, specifically Vol. 1's chapter on the British West Indies)
African slavery wasn't new by the time that the British sugar islands of the 17th century began using them instead of indentured servants. Cuba had slave ships come in possibly as early as the 1520s (Roland Ely: Cuando Reinaba su Majestad el Azucar). It also had the disadvantage of being pricier than indentured servitude. But it offered a dedicated workforce which 1) wouldn't be compensated with land of their own, 2) could be worked and abused for however long planters wanted, and 3) could be brought to work without any positive incentives.
Once African slavery became a reality, it was a question of justifying this reality with some kind of rationale. In this case, racism. You can find multiple in-depth analyses of how racism as a belief system came to be in these threads from our FAQ section in the sidebar.
I think that Aimé Césaire had it right when he stated in his Discourse on Colonialism, that colonies are political and socio-economic systems but that racism is the belief system which often held them together. Black labor was useful, therefore it had to be justified. Remember, we're dealing with human beings here. Even slavers need to believe they're not bad people. They needed black people to be subhuman in order to justify their brutal exploitation. As time went on, pre-scientific prejudices and generalizations became pseudo-scientific prejudices and generalizations. The transformation of black people into subhuman fodder for the fields went apace with the advance of industrialization in agriculture.
Sven Beckert (Empire of Cotton) and Edward Baptist (The Half Has Never Been Told: Slavery and the Making of American Capitalism) have both talked about how the advance of technology, the cotton gin in the case of their topics, upped the demands placed on black labor. In the case of sugar this also applies, with advances in sugar mills and refineries allowing for larger and larger amounts of cane to be processed, leading to greater pressure and abuse being heaped upon workers.
There's no evidence black people were naturally inferior in intelligence or magically able to work the fields without being harmed. Being darker skinned may have given some kind of edge against sunburn compared to light skinned people, but hats exist for a reason. As do clothes to cover up. Black people suffered from long periods of work under the heat of the sun too. They also got sunstroke. They also got sunburn. They are also reasoning, thinking, feeling human beings. The horror of treating fellow human beings like animals was what necessitated the lies of racism. Racism is how planters learned to live with themselves and what they did.
I hope this at least partially answers your question.