There's actually a Swedish-speaking village in Ukraine, called Gammelsvenskby (Old Swedish village) made up from Swedish-speaking Estonians that were deported there from Estonia during the reign of Catherine the Great. However, that is still in Europe, so I think it does not count from your question.

The Swedish Gold Coast colony was shortlived, as was Nya Sverige in America. As /u/hrimfrost say, the shortlived status of Swedish colonies and the lack of attempts to make Saint-Barthélemy Swedish-speaking means that there is little legacy left of what little there was.

The reason for the lukewarm Swedish enthusiasm for far-flung colonies were the need for a large army (which meant the navy had less resources), the need to pass both the Dutch and the English on the way out into the Atlantic (both whom, were jelously guarding their own colonial ambitions) and the immediate and closeby tolls of the Baltic Sea (and North Sea) ports which provided a lot of profit for the Swedish state.

By colonial holdings in the Caribbean, I assume you mean Saint-Barthélemy? Sweden held it between 1784 and 1878, I believe. It was sold back to France at that point, because it had ceased being profitable for the Swedes - apparently, there was a referendum, and only one vote was cast in favour of staying Swedish.

As for remnants of the Swedish era on the island, there isn't much. There's one town named Gustavia, after Gustav III of Sweden, and Gustavia has some Swedish-sounding street names and such, but it is culturally mainly French. This is likely a consequence of there never having been a push to make Swedish the main language of the island - even when it was in Swedish possession, laws were written in English and French, and those were the main languages spoken as well.

Laws regulating the daily lives of the black/creole population of Saint-Barthélemy during the Swedish rule