I think you have a flaw in your basic premise. The Umayyad Caliphate (the aforementioned Muslims) who conquered Spain DID have an empire in North Africa, but they were culturally Arabs.

If the Umayyad count, then I guess the Ottoman Empire does too. They had holdings in N Africa and Europe simultaneously.

After Carthage was destroyed by Rome in the Third Punic war, North Africa was under the control of the Roman Republic / Roman Empire until the Vandals overran Tunisia. The Vandals sacked Rome in 455 AD, but they were a fairly short lived Barbarian kingdom. They were destroyed by the Emperor Justinian rather easily, by 533 AD. Before Carthage, the only well known North African civilization is the ancient Egyptians. During their New Kingdom, they expanded as far north as Syria, but there is no evidence that the ancient Egyptians tried to expand into Europe.
There was one Roman Emperor, Septimius Severus, who was born in Leptis Magna, a Roman city that is in the modern nation of Libya. However, Septimius Severus did not become powerful until the Roman Emperor Comodius appointed him the commander of a legion in Pannonia, on the Danube river frontier. Septimius Severus used this legion to become the Emperor of Rome, in 193 AD, after the year of five emperors. However, his power base was not in North Africa, but rather the middle of the Danube Basin.
Sources: "The Severans: The changed Roman Empire" by Michael Grant 1996
"The Vandals" by Andy Merrills and Richard Miles 2010
Andy Merrills also wrote several articles in scholarly journals about Berbers, Vandals and Romans in late antiquity in North Africa.