I'm going to steer clear of "regional backwater" and note that there's a fair bit of talk about why North Africa was so prosperous under Rome and less so in the centuries to follow. In his new book Peter Heather (Restoration of Rome, Oxford UP 2014) makes the argument that the spectacular agricultural prosperity of Roman North Africa was a product of the unique position it held inside the trade networks of the western Mediterranean, and how so much of its trade was done with and by the Roman state itself. Certainly, North Africa continued to be prosperous and a source of agricultural wealth long after the end of Roman (or Byzantine) rule, but with its big customer (the state) gone, that prosperity was muted. As Heather puts it (pp. 171ff.): "But North Africa's Roman prosperity had come from the fact that it was tied into a broader system of west Mediterranean exchange which was actually dependent in a series of ways on the West Roman state, not least because it [= the roman state subsidized transport costs [of NA agricultural products] for its own purposes...It [= NA] exported [after the Roman period], therefore, but on a much smaller scale, and the general level of wealth in the region seems to have settled back into a more modest prosperity." This argument isn't new in this book--you can track down for where Heather got it from his notes.

This is an interesting question and one that historians have hotly debated for the better part of one hundred and fifty years. My expertise is on North Africa during the central Middle Ages, so this answer will focus on developments in the 10th, 11th, and 12th centuries, although I am sure there is some fascinating research on the role of North Africa in the centuries directly following the Islamic conquests. This answer will also focus on “Ifriqiya,” which comprises the modern-day areas of Tunisia, western Libya, and eastern Algeria.

First off, I think it would be wrong to suppose that North Africa was simply a “regional backwater.” As Michael Brett has argued, North Africa was a crucial part of Mediterranean trade during the central Middle Ages, both for routes heading west from Fustat and Alexandria, as well as being a supply of gold from sub-Saharan Africa (Brett, "Ifriqiya as a Market"). The centrality of the area of Ifriqiya in the Mediterranean meant that it was an incredibly important hub for overseas and inland trade during the high Middle Ages. S.D. Goitein went so far as to describe the area of modern-day Tunisia as the "Hub of the Mediterranean" during the tenth and eleventh centuries (Goitein, "Medieval Tunisia: Hub of the Mediterranean"). In particular, Mahdia and Qayrawan served as major hubs in the sub-Saharan gold trade, which they traded for Sicilian grain. Qayrawan, whose name derives from the word "caravan," was a center for land-based trade that spanned across North Africa as well as parts of sub-Saharan Africa. Mahdia, acting as Qayrawan's port, was a hub for primarily maritime commerce.

However, one of the traditional turning points for the history of North Africa comes in the form of the Banu Hilal, an Arab tribe that was supposedly let loose by the Fatimid rulers of Egypt in the middle of the 11th century as punishment for the regional rulers of North Africa declaring their independence. Two distinct historiographical schools have formed with regard to the Banu Hilal. The first, led by French historians of the early-mid twentieth century, argues that the economy of Ifriqiya was prosperous in the tenth and eleventh centuries (see citation group #2 on attached comment). To them, the invasion of the Banu Hilal in 1057 brought incredible devastation and destruction to the region, which would devolve into a series of ethnic conflicts between the Sanhaja and Zenata Berber tribes. However, a second historiographical school considers the invasions of the Banu Hilal in a different light. This perspective, adopted by most historians from the 1970s onwards, argues that the economy of Ifriqiya began to decline following the Fatimid relocation of their capital from Tunisia to Egypt in 969. Various groups competed with each other over control of key trade routes and the Banu Hilal were merely one of these many groups that sought economic control. At worst, the Hilalian invasions sped up an economic process that was already well underway. Regardless of these two historiographical schools, it is generally accepted that there was decline in Ifriqiya following the Hilalian Invasions of the mid-11th century.

As for /u/weltburger’s comment that Europe was simply burning witches, this is a gross oversimplification. During the 11th and 12th centuries, there was an intricate relationship between the powers on the Italian peninsula and the regional Muslim rulers of North Africa. Following the Hilalian invasions, power in North Africa became increasingly fragmented and subject to incursions. A Pisan-Genoese coalition sacked Mahdia and its suburb of Zawila in 1087 and the Normans, based in Sicily, seized multiple cities along the African coast during the middle of the 12th century (Cowdrey, "The Mahdia Campaign of 1087"). Even before the Norman invasions, the regional rulers of North Africa had been forced to accept unfavorable treaties with the Normans in order to ensure that they could feed their people. A series of devastating droughts meant that the rulers of North Africa were reliant upon Sicilian grain, a dramatic reversal of the economic situation in ancient Rome.

Few people here have directly answered the OP. Here is an excerpt from a review of Diana Davis's Resurrecting the Granary of Rome I completed in grad school:

Davis argues that after conquering Algiers in 1830, French colonial authorities crafted a declensionist environmental history of Algeria. This narrative related how eleventh-century Arabic invaders and pastoral nomads had squandered the once-fertile breadbasket of Rome. This contrived history reflected French racial assumptions (depicted creatively by Davis through full-color pages of contemporary French art) but also immediate military need for lumber. It legitimized lucrative French resource extraction and in turn delegitimized extant Algerian practices of pastoralism, collective ownership, and controlled burns. Davis finds that, curiously, none of the narrative was based in fact. Classical historians who described lush environs had often never actually visited North Africa themselves. Ruins of the Roman Empire had given the notion archaeological evidence: abandoned aqueducts, olive presses, and cities supposedly exhibited destructive or at least lackadaisical Algerian primitivism. Davis proves instead the topography actually changed little over the last two thousand years and elaborates the ultimate irony of uninformed, detrimental French land conservation efforts. Prohibiting prescribed burns dampened growth as newly introduced, water-intensive vegetation sapped soils across the region.

...

Distinctly concerned with the present, Davis documents (and obviously resents) the ongoing trope of independent Africa's decline. Desiccation and impoverishment are not new developments but instead the festering legacies of colonialism: the former from introducing eucalyptus, citrus, and other absorptive plants; the latter from local economic disfranchisement and wealth extraction.

Part of your perception may be because the 'action', so to say, of the Roman Empire was in the North with Germania, and the East with Parthia.

North Africa was still very wealthy and vitally important to Rome even during it's fall in the 400s AD. When the Vandals took over North Africa, Rome lost access to an annual crop of half a million bushels of wheat, most of its olive and and a large portion of its tax base, as Carthage was the second largest city in the Western Empire (Terry Jones, Barbarians, 2006). It was also prosperous enough to equip and house the Vandals, to the point where they could mount raids and expeditions into Italy, Sicily and such, sacking Rome, and later defeating the Romans when they tried to retake the province.

When the Arabs came to take Africa from the Byzantines, they came back from their raids with lots of loot. In 647, an Arab army marched into southern Tunisia, defeated a Byzantine army, and after looting metals, women, and such, returned to Egypt. A sign of how much loot they returned with is shown by the bonus the troops received: 3,000 dinars for cavalrymen, and 1,000 for infantry. The army that completed this raid is estimated to have been around 9,000-10,000 men when it left Egypt. (Antonio Santosuosso, Barbarians, Marauders and Infidels, 2004)

I've heard the land there used to be much more fertile. Does anyone have any sources to confirm this?

I'm surprised that I don't see anyone mentioning the significance of Tunis, Algeria, Libya, and Morocco being periphery states of the Ottoman Empire. Under the protective umbrella of Ottoman identity the Barbary Coast flourished to some degree by bullying wealthy nations such as England and Spain into paying increasingly large sums of tribute money which granted their ships relative immunity from their pirate activity, while avoiding any real conflict with these states, and more importantly, the larger Empire to which they belonged. By the late 18th century the ties between Constantinople and the Barbary Coast were dwindling and a number of nations including Portugal and later the United States would change the status quo within the Mediterranean trade networks by launching campaigns against the Barbary pirates. These were largely effective and were the beginning of the end for an irreplaceable source of resources and wealth in a region where there was and is very little of value geographically speaking. My experience with this subject is pretty limited, but I'm getting this from a History of the Ottoman Empire class, Karen Barkey's book Empire of Difference, and my more thorough knowledge of the Barbary Coast Wars.

How much of this alleged provincialism is a product of 19th century imperialism, when these places were either satellite tributaries or literal provinces?