Do you mean the modern Olympics, which started in 1896? At that time the only African countries not ruled by Europeans were Ethiopia, Liberia and Egypt (which might be considered a British puppet government). The only non-European competitors were from the US and the Australian colonies, but I've never seen anything to suggest that other countries would have been barred from entering. The first few modern Olympics were tiny events compared to modern games. It's hard to imagine that anyone outside the comparitively small world of 'western' sports afficionados would have even been aware of their existence.