Simply put: Because Roman Africa was very different from modern North Africa, and your impressions are mistaken.

Contrary to what you assume, Northern Africa in Roman times was immensely imported economically and financially. In many ways, it was the heart of the Roman economy in the Western empire.

The giant grain-fleets of the Annona sailed from the coasts of Africa to Rome yearly, feeding the million+ inhabitants of that ancient metropolis. Grain was also produced in Sicily and Sardinia, but Africa was the main source for the West. (Egypt fed the grand cities of the East, although before Constantinople was founded Egyptian grain also went to Rome.)

The shipmasters of the grain fleets were required to transport the grain by the government, in return for a remission of taxes, and were also allowed to transport as many other goods as they could fit in, and continue to trade tax-free for up to several years afterwards. This was a huge boost to the Roman economy in general, and the African one in particular. With the grain fleets already sailing across the Mediterranean, it was an easy to thing to also transport masses of African red slip ware ceramics, which is found all across the Roman world, even in poor and rural areas. The ships could travel to Spain to load olive oil and wine, trade across the coast of France and Italy and all the islands in between, and their goods would continue farther still inland.

Carthage itself, meanwhile, was rebuild by the Romans and soon became a metropolis again, as prosperous and wealthy and influential as it had been during its heyday in the Punic wars, although not on a political or military level. Of the mass of towns and cities in the Roman empire, only 5 truly stood out among the crowd, reaching into the hundreds of thousands of inhabitants. These were: Rome itself, Antioch, Alexandria, Constantinople, and Carthage.

Culturally, it was as important as it was economically. The last great dynasty of the Principate, the Severans, (Septimius Severus, Caracalla, Geta, Elagabalus, Severus Alexander) hailed from Africa. The great Augustine of Hippo hailed from, well, Hippo Regius, which was a city in Numidia, modern day Algeria.

Indeed, such was the importance of Africa to the Roman west, that most modern scholarship holds that the death-blow to the Western Roman Empire as a political union was either the loss of Carthage to the Vandals, or the failure of the subsequent Roman attempts to retake it. (Vast military operations by land and see drawing on the combined resources of the Eastern and Western empires, whose failures were disasters as big or bigger than Adrianople or the battle of the Frigidus.)

Guy Halsal even describes the subsequent war between Rome and the Vandals, only half-joking, as "the fourth Punic war," in which the Vandals used their control of the African fleets to make war on Italy and even sack Rome. (Although, interestingly enough, the grain trade continued for a while even after Rome lost control of Africa, albeit by private merchants instead of the government-sponsored grain fleets.)

The differences are partially explained by climate change: 2000 years ago, Africa was more fertile than it is now. There were also great works of irrigation, such as demonstrated by the fascinating story of the Garamantes, a desert dwelling North African people who used underground canals to sustain a city in the middle of the desert.

Even then, Africa in Roman times was a region with many differences and contradictions. The coastal cities and villas could be immensely rich, important and productive, but much of the hinterland was indeed poor and desert-like. The Berber tribes were an ever-present danger. Religious schisms between orthodox and Donatist Christians troubled the region in late antiquity, and when the Arian Vandals took over in the 5th century this did not help anything.

All in all, though, Africa's importance to the Roman Empire was immense. It should come as no surprise that they occupied it for so long, and tried to re-conquer it so desperately when they lost it.

Sources:

• Guy Halsal, Barbarian Migrations and the Roman West

• Walter Scheidel, A comparative perspective on the determinants of the scale and productivity of maritime trade in the Roman Mediterranean