So, with this (hopefully correct) 'beginning' and 'end' point of Malian history established, what fills the middle?

Be careful there.

I find that this is a thing that tends to happen a lot with places on the map in Africa. The modern state of Mauritania is not the same as the ancient Berber kingdom that fought the Romans. The modern state of Ghana is not the same thing as the 11th century kingdom along the Niger and Senegal rivers. There is a region in Somalia called Puntland, but we don't know where the kingdom of Punt that Hatshepsut traded with was.

By the same token, the Mali Empire of the 1320s under Mansa Kankan Musa is not the same thing as the 20th century state whose capital is at Bamako. There is not an unbroken political lineage from the one to the other.

And even talking about the gap from Musa's hajj to Cairo and Mecca in 1323-24 to the onset of French military annexation of West Africa in the 1870s, that is a gap of 550 years. There is an enormous amount of time where political, economic and social changes occurred. I'll try and give a thumbnail sketch of some important events that happened over that period, but I expect I'll miss a lot.

Ok, step one is we are gonna skip ahead to about 140 years after Musa's hajj to the 1460s. There is a revolt in the city of Gao which, along with Niani and Timbuktu is one of the three largest and most important cities in the Mali Empire. Gao had been the capital of an independent kingdom called Gao (or Kawkaw) before being conquered and brought into the Mali empire, and a distant descendant (Sonni Ali) from the royal lineage of the Gao kingdom led this revolt. Sonni Ali was extremely successful, and he ended up establishing an independent kingdom centered around Gao, called the Songhai empire. But, he wasn't content to just be independent from Mali, he sensed that the old empire was weak, and he and his successor Askia Mohammed I managed to conquer much of the territory of the old Mali empire, making the Songhai empire almost as big as the Mali empire was at its height. Now, Mali held on as a distinct political entity, but at first it was a much-reduced rump state, and at times it was forced to pay tribute to Gao.

Separately, around this same period in the 1460s, the Portuguese were sending ships down the Atlantic coast of Africa and were beginning to trade with the peoples of the coast. At first, this trade took the form of trading African cloth for European cloth, or Kola nuts, ivory and copper for glass, pig iron. Eventually though, Portuguese traders and later Dutch, English, French, Danish and other European traders came to be less interested in cloth and copper, and instead came to trade for enslaved people, who would be transported to plantations in the Canary Islands, Sao Tome, and Portugal, and in the later 1500s would start to be brought to the Caribbean and mainland North and South America.

This development of legitimate trade, and then the transformation into the Atlantic slave trade, is important because it fundamentally redirected the existing network of trade in West Africa. In the time of Musa, the trade networks began at the headwaters of the Senegal and Niger rivers in the south, and headed north into the sahara, bound for Morocco, Tunisia and Egypt. By controlling the Bambuk and Bure gold fields as well as major trade entrepots like Timbuktu, Gao, Awadaghust and Walata, the emperors of Mali (and of Songhai) had enormous control over the trade in gold and in slaves across the Sahara, making them rich.

With the opening of new trade along the Atlantic coast, and the intensification of trade (in slaves, but also in gold, spices, ivory) from the 15th to the 19th centuries, there was now an alternate trade route into and out of West Africa for goods and people. This undercut the commercial position of the Songhai Emperors.

So, skipping to 1590, the Emperor of Songhai is in a political dispute with the Sultan of Morocco over who owns the oasis town of Taghaza. It is right in the middle of the desert, halfway between Timbuktu and Fes, and it is an important source of salt, which is a valuable commodity at the time.

Neither side is willing to back down in the dispute, so the sultan of Morocco sends an army of musket soldiers to cross the desert and strike at Timbuktu. In the ~130 years since the great revolt against Mali, the Emperors of Songhai had prided themselves on the strength of their cavalry, and had not been keen to adopt firearms. So, when this Moroccan army reached the heartland of the Songhai empire along the banks of the Niger river, you had a clash of a gun-wielding army against a crush of horsemen at the battle of Tondibi. Long story short, the Moroccans were victorious and managed quickly to capture Djenne, Timbuktu and Gao, but got mired down fighting the much diminished Songhai state.

So, in the 1600s you have political fragmentation in the West African savanna, with the Moroccans holding on to Timbuktu and Gao, but lots of other areas formerly in the Songhai empire going their on way. In this era you get the Wolof kingdoms in Senegal, you get the Fulani states like Futa Tooro, Futa Jallon. Around this same time the Bamana (called Bambara by the French) are in the area that is now Bamako and Mopti, and they start to establish an extensive kingdom centered around Segou under the king Biton Koulibaly around 1700. That kingdom of Segou, also called the Bamana empire, ends up controlling a stretch of the upper Niger river that is roughly the southern half of the modern Republic of Mali.

Skipping ahead again, starting around 1775 and continuing into the early 1800s there is a political-religious revolution among the Fulani (or Fulbe, or Peul) people of West Africa. In Futa Tooro, and in Futa Jallon, there are revolutions which focus on the purifying of Islam in the realms. Fairly soon, among Fulani further east there are similar revolutions and conquests, and the Bamana state at Segou gets conquered and replaced by a Fulani-led kingdom with a capital at Macina. Even further east, Usman Dan Fodio overthrows the several Hausa kingdoms and establishes the Sokoto Caliphate circa 1800.

After that, around 1850, another would-be religious reformer in Futa Jallon named Umar Tal begins organizing his followers and creating his new empire. He conquers Macina, Kaarta, and various minor Bamana kingdoms, and his empire gets remembered as the Tukolor (or Tucoleur) empire.

But, the French are already making alliances and pushing military expeditions up the Senegal river in the late 1850s. A clash doesn't happen right away, but a generation later in the 1880s, as European competition for empire leads to a rush to claim territory across the continent, French forces ally with disgruntled Bamana subjects and conquer the Tukolor state in 1890 and take control of the area that is now the Republic of Mali.

---

So, yeah, from 1320 to 1890 there was more than 500 years of kingdoms growing and being conquered, of trade routes changing, and of the growth and decline of the Atlantic slave trade.

I don't want to dismiss out-of-hand the idea that French colonialism did cause societal and economic harm to the societies in what is now the Republic of Mali. That is also a factor for how Mali got to its current levels of poverty. But, that region in 1850 was not as united, powerful (relatively) or wealthy (relatively) as it was 500 years earlier.