The Jews did hold slaves themselves and the story of Ham was a mythological justification of the enslavement of Canaanites^1 .

Modern Christian slavers added only one particular detail to that myth: Ham was made brown^2 . This name (חם) was conveniently similar to the Hebrew word for «brown» - «hum» (חוּם).

This identification of «brown people» with the «curse of Ham» indeed would live on amongst the manumitted slaves and their own Christian organisations.

source:

^1 Genesis 9:20-27

^2 McKee Evans, William. From the Land of Canaan to the Land of Guinea: The Strange Odyssey of the Sons of Ham. American Historical Review 85. 1980. P.15-43.