According to Frederick Douglass slaves would often be encouraged to fight and wrestle for sport because it’d keep them tired and fight off the “spirit of insurrection.” This led to some level of organized boxing matches and gambling rings between slaves. In 1810, Tom Molineaux, after he earned his freedom and $100,000 for his former master in a fight against another slave, fought for the Heavyweight Title in London.

Actual historians with a depth of knowledge could add a lot more context and examples but I think it was something slave owners were generally ok with and profiting from in the first place, and it became a sensational thing in the rest of the anglosphere decades before the civil war. It probably drew bigger crowds to see Africans or African-American Freedmen fight just for the novelty factor, it likely didn’t anger abolitionists, and it was making slavers a bunch of money.

Sports like baseball developed later in the reconstruction and segregation eras and leagues and organizations built around those policies, while black people had been grandfathered into boxing for generations. At least in sensational cases it was a way for them to literally “fight for freedom.”