Why is that?

The Sahara.

Without long-range navigation maintaining an empire across the sahara would be forbiddingly difficult, without even considering the difficulty of conquest: you'd have to bring an army through ~2000km of desert, with few resources you didn't bring with you (none if you couldn't get the saharan nomads on your side) and on the other side you'd reach settled lands unlikely to appreciate your incursion.

It was much simpler and cheaper to control and protect the trade routes between sub-saharan africa and the mediterranean, doing that was part and parcel of Phoenician and Carthaginian rise, Rome did that afterwards (mostly through Legio III Augusta).

As well as the desert, one of the barriers to expansion by invaders into sub-Saharan was disease. There is a theory (and I can't for the life of me find a link to it online, and would be eternally grateful to someone who could provide it) that sleeping sickness acted as a barrier to the Muslim conquests extending beyond the Sahel, due to the fact it killed off their horses.

Edit: Apparently it can be found in Eliza Griswold's The Tenth Parallel: Dispatches from the Fault Line Between Christianity and Islam