The short answer is that it's because language changes involve more than who controls what territory. The Roman Empire dominated the Mediterranean basin, but the Latin language only really came to dominate the western half of the empire. Why? Partly that was patterns of conquest, immigration, and cultural assimilation practised in, for example, Gaul. Partly it's due to the post-Empire history of Western Europe.

The situation in the eastern half is different, because of both its history prior to conquest and assimilation into Rome, and its history after the fall of the western half of the empire. The successful spread of Hellenic language and culture, and especially the far-reaching impact of the conquest of Alexander which essentially made Greek culture the dominant elite culture throughout the western mediterranean. This didn't change that much under Roman rule, and indeed, Greek language and culture functioned as an elitist culture for Roman elites, while still continuing to be the language of most of the cities in Greece, Asia Minor, and Egypt. This obviously continued into the Byzantine period.