I don't know about Britain. The rest of the Empire has to be divided into two parts: the Eastern Roman Empire, and North Africa. Latin was spoken in North Africa, up to but not including Egypt - it was spoken side by side (influencing, and being influenced by) the local Berber languages, as Latin languages were mixed everywhere, but it was recognizably Latin, especially by the coast and the further North you stayed.

The Eastern Roman Empire we can divide into a number of parts: Anatolia (or Asia Minor, modern-day Turkey), the Near and Middle East (or the Levant), and Egypt. Neither of these areas were Latin-speaking - Anatolia was Greek speaking, the Levant was Syriac-Aramaic and Greek in the cities, and Egypt was Coptic (aka Egyptian) and Greek in the cities.

What happens here is, of course, the Arab invasions. These spread towards the East, to Iran and North India, and to the West, towards the Levant and reaching the Iberian Peninsula. I think a more productive way to frame your question would be to look at it not from the point of view of the Roman Empire, but from the point of view of the influence of Arabic as a language. William Thackston remarks that "Arabic never displaced a language to which it was not related". Is he right? Both the Berber languages and Coptic are Afro-Asiatic languages, which makes them related to the Semitic languages; Syrian-Aramaic is Northwest Semitic. On the other hand, neither Iberian Latin, Greek, or Middle Persian are related to Arabic in anyway. It seems he was right.

Now, by the time of the Arabic invasions there were no Celtic languages spoken anywhere in the Iberian peninsula. Latin, or Hispanic Latin if you want to be precise, had completely replaced the Celtic and Iberian languages that had previously flourished 8 centuries before. (Basque, a language isolate, is the great enigma here). Arabic did not replace Latin, which by this time was starting to turn into what we call "Romance", the precursor the the Romance languages of the Pensinsula, however much it influenced it. In North Africa, it by and large replaced the Berber languages where they were spoken side by side with Latin (though not the farther South you went). There were still speakers of African Latin in the second millenium, even if Arabic was well on its way towards linguistic hegemony, and even to this day a very large percentage of the population still speaks various Berber languages.

Now you may ask, why did Arabic swap Latin, since so many of these lands were already under the control of Germanic barbarian kingdoms? Most of these lost their language fairly quickly and assimilated upon settling into the Empire. They Romanized, as people say, which even if it is a complex notion can be said to refer to the way barbarian peoples took up the customs, lived side-by-side with their Roman subjects - even when they tried to keep their lawcodes distinct, the end-result was, invariably, assimilation. The Arab conquests were very different. Maybe out of an attempt to preserve religious exclusivism, since Islam was not a proselytist religion at the beginning, Arabic armies lived either in independently founded cities, or else in garrisons outside of larger cities. They did not receive lands from their subjects to settle in (who, by not being expropriated, saw their conquerors with kinder eyes), and were paid by the central authorities of the caliphate. They did not mingle with the conquered peoples. Hence they did not lose their Arabic language, which in time, as it grew in influence (and especially from the moment when it became compulsory to know Arabic to serve in state offices), made the population more and more turn to it in detriment of their previous languages.

Why did the populations in these other places, such as Iran and the Iberian peninsula, did not "turn"? Obviously Arabic had a huge influence in both the Romance languages, composing 10% percent of Spanish and 11% of Portuguese vocabulary; in Farsi I've seen percentages up to 1/3 of the vocabulary being of Arabic origin. Yet, crucially, to complement the "never displaced an unrelated language", political control counted for much. Agitation in Iran started from a very early stage (being a main reason for the so-called ʿAbbasid revolution), and through the 9th and 10th centuries Iran was ruled by the Samanids, that helped push the Iranian language from the brink of oblivion. In the Iberian peninsula there's of course the Reconquista, which pushed in a North->South direction (and that is precisely the order of "Arabisms" present in both European Portuguese and Spanish, with many even considering Andalusian, a highly Arabic-influenced dialect of Castillian, as a different language in its own right.