There is more of an African influence in the language in the Caribbean for a few reason: First, the Caribbean sugar plantation was extremely deadly compared to the cotton and tobacco they were growing in the US. Therein, people lived a lot longer and their children survived more in the US so many less enslaved people on plantations had actually been born in their homeland in Sub-Saharan Africa. Therefore most of the enslaved people in the Caribbean were African and the US enslaved people where African-American (in that they were born in the US). Second, there were more places to escape to in the jungles of the Caribbean and the government there was less in control. This lead to escapees creating communities of their own outside the plantation, often joining native people who had retreated to the jungles. This lead to significantly more language exchange than the limited contact enslaved people had with native groups in the US during slavery. Third, there were many less white people in the Caribbean and also less enslaved people working in white homes, so there was less contact with white people, making it easier to preserve languages among enslaved people because children would spend more time around adults who were more free to speak their own language because they were in the sugar cane fields without much supervision. I would say the first reason is probably the most important but all three contributed.

Your premise is a bit off. This post about Gullah culture from /u/thechance might be of interest to you.