My question is why did they leave so little behind?

Partially, this has to do with the recovery of archaeological sites. Italy and Egypt have been picked over by archaeologists and antiquarians for nearly 500 years, so there is an extensive archaeological and architectural record nearly everywhere in those countries. The discipline of archaeology was pretty much born in those places, where people focused on trying to understand the history of the region through the material and architectural remains of past societies. Even today, people find new sites in Italy and Egypt all the time! The scale of the task of sampling in archaeology is truly massive. Italy and Egypt are relatively large countries, but a region the size of West Africa is many, many, many times larger than those two countries put together, and archaeological research there has only been going on in earnest since maybe the 1930s, and at a much lower intensity than in places like Italy or Egypt. I can maybe count the number of archaeologists I know who work in West Africa on one hand. In comparison, today there are perhaps hundreds of archaeologists working in Italy alone.

The other thing I want to address is maybe a bias you have for what constitutes civilization and "greatness", as you put it. This is a relatively common framework that pretty much everyone learns in school and just through normal public discourse, which then has to be unlearned if you want to take a scientific approach to studying society. The idea of a society being "great" is pretty much entirely subjective, and is based on a specific point of view of whoever wrote about that society. The development of a society can't be measured or compared on the basis of architectural or artistic development, because there isn't actually any standard model for the way that large-scale societies organized and developed in the past. It makes sense to chart and map the development of architecture and art within a certain context, but once you start trying to compare and stack up different societies within the same, unilinear framework, it becomes impossible to remain scientific. You end up with what amounts to a set of moral judgements. This was quite useful if you were a British imperialist trying to justify the conquest of East Africa in 1885, but less useful if you're a modern scientist trying to produce accurate and meaningful descriptions of social organization.

To sum up, historical societies organized themselves in diverse ways, and some forms of social complexity produced massive architectural features, while others didn't. You can't deduce social complexity or the power of a state from solely studying the architecture and art alone. There's just no one-to-one relationship, because humans aren't ants and societies can't be modeled in laboratory conditions.

Let's be clear that there are in fact, massive monumental structures all throughout sub-Saharan Africa. The longest and largest earthworks in the world are found in Nigeria, the walls of Benin. The Swahili coast of East Africa is full of monumental architecture in stone-built towns, built starting around the 12th century. Ethiopia has numerous large monumental churches. It's there if you look for it, it's just not well researched and not well publicized. I recommend the PBS documentary by Henry Louis Gates, Africa's Great Civilizations. But I would keep the above critique in mind-- this documentary is really well made and has a lot of great footage, but it does sort of reinforce the trope that social complexity = large monumental architecture. For every society that produced a large monumental structure, there are others which did not, but instead left more ephemeral sites. These latter societies are equally interesting and important from a social scientific point of view and shouldn't be overlooked.

As for your question about Zimbabwe and southern Africa, its not true that Great Zimbabwe and Khami were the only large stone-built towns. These sites were just especially wealthy, based on their control of the gold and ivory trade to the East African coast. But there are possibly hundreds of smaller stone-built settlements throughout southern Africa.

One final thing I'd like to stress-- the lack of monumental architecture doesn't necessarily have much to do with the availability of building materials. When a society wanted to build monuments and had the labor power, they found a way. Southern Mesopotamia has almost no stone at all, or even trees, or virtually any natural resources, and yet some of the oldest and largest structures on earth were built there.