Well, you are sort of getting into a bit of a methodological issue here. How do you know if something is an empire if you don't have written records? If you have studied any Chinese archaeology, it is a similar problem one might have with interpreting Erlitou/Xia and Erligang/Shang.
That being said, the Niger seems to have hosted a fairly impressive level of urbanization that was roughly temporally equivalent with the classical Mediterranean, and there are also ancient examples of widespread material cultures, such as Nok in northern Nigeria. Actually leaping from these to the imperial designation is probably not possible, certainly with the extremely limited archaeological exploration of the region.
The earliest Arabic sources mention Ghana, as well as other sahelian states like Gao and Takrur.
Take for example this description from the Tarikh ibn Wadih by al-Yaqubi in the 9th century.^1
There is the kingdom of the Kawkaw, which is the greatest of the realms of the Sūdān, the most important and most powerful. All the kingdoms obey its king. Al-Kawkaw is the name of the town. Besides this there are a number of kingdoms of which the rulers pay allegiance to him and acknowledge his sovereignty, although they are kings in their own lands.
Incidentally, for a time Takrur captured the imagination in the medieval Arab world on account of the sultan of Takrur's reputation as a zealous convert to Islam.^2
The reason that Ghana outshines the contemporary states in popular memory would seem to be the human obsession with gold. The Ghana empire and its ruler came to be associated with extreme wealth in gold in the writings of medieval Arab geographers, while other states did not enjoy this reputation.
Susan Keech McIntosh provided a thorough reassessment of what processes may have led to the development of these states in the mid-first millenium in her article Reconceptualizing Early Ghana. It's well worth a read.
1- Corpus of Early Arabic Sources for West African History by Nehmia Levtzion and J.F.P Hopkins. pp. 7
2- The History of Islam in Africa by Nehmia Levtzion and Randall Pouwells. pp. 64