As recently as the early 7th century Greek speakers ruled almost the entire eastern Meditteranean. Successive waves of conquest pushed their leadership further and further out. They lost the holy land and north Africa in the 700s, and began to lose Asia Minor(Anatolia) with the battle of Manzikert in 1071. The Ottoman Turks finally completed the conquest of the Greek speaking Byzantine empire in 1453. At that point, all that was left under Greek rulers was the city of Constantinople itself and a couple other small outposts. Most of the area of modern Greece was taken by the Turks before 1453. Individual islands held out for longer, but those were usually under Italian rulers.

In the initial centuries after these conquests, Greek remained an important administrative language in the conquered territories. This died out in a few centuries under the Arabs, less quickly under the Ottomans. Greeks flourished throughout the period of the Ottoman Empire as an administrative and commercial class. Greek governors ruled over large tracts of Eastern Europe for the Ottoman sultans. At the beginning of the 20th century there were large and wealthy greek populations throughout the eastern Mediterranean, most famously in Izmir on the Turkish Coast, and in Alexandria in Egypt.

This population that had survived and even thrived for centuries, could not survive 19th and 20th century ideas of nationalism however. After Greece won its independence from the Ottomans in 1832, the Greek population lost its privileged position in the government. They managed to stay dominant in the commercial activity of the Empire however. In the early 1920s Greece tried to take advantage of the Ottoman Empire's loss of the first World War to re-conquer lands Greeks had not ruled for over 450 years. They failed miserably. The Treaty of Lausanne provided for the exchange of populations, around 1.5 million Greeks were ejected from Turkey, and about 500,000 Muslims were ejected from Greece. This ended 2500 years of Greek presence on the eastern coast of the Aegean sea. A large Greek population survived in Istanbul until the 1950s, when race riots and the continued persecution by the Turkish government forced most of the rest out. Egyptian President Gamal Abdel Nasser kicked the Greeks out of Alexandria in the 1950s as well.

Greek was not the Eastern Latin. Hopefully, a specialist of the Hellenistic and Roman era will be able to give a more in-depth answer, and explain why it never took root so efficiently, but it is pretty clear that by the time of Late Roman Empire (6th century), various languages cohabited in the Eastern Mediterranean. The population of Egypt spoke Coptic, a distant evolution of ancient Egyptian (still surviving as a liturgic language); the population of Syria used a language aptly named Syriac, a version of Aramaic (a Semitic language; it may have had a tenuous survival as an oral dialect, but I am not as sure about it). All these languages were endowed with (relatively recent by the 6th century) vernacular literary traditions. Other Anatolian languages may have had survived to this point (like Phrygian for instance); however, it is probable that they were also on the wane, and gradually replaced by Greek. So all in all, the only place where the disappearence of Greek is surprising could be Anatolia.

That being said, at least two elements mitigate this impression. The first one is that large parts of Anatolia had been under intermittent foreign control for centuries when Constantinople eventually fell (a possible example: the Sultanate of Rum). Certainly, a degree of bilinguism with the languages of the occupiers must have been common by that time. The second one is that you may be underestimating the number of self-identified Greeks or Romans living in what is now called Turkey in the beginning of the 20th century: more than one million Greeks lived in Anatolia before WW1, many of which were either killed, or expulsed to Greece.

This is mainly due to the fact that after the byzantines, the East was dominated by populations with a culture completely different form the Hellenistic one. With the Arab War that started in the 7th century, Byzantium lost Egypt and most of its Syrians holdings to the Arabs. Then came the Seljuk dinasty from the East, which occupied center Anatolia. By the 12th century, Byzantines' holdings in Asia amounted to western Anatolia and the coastline. By 1300, when the Ottomans appeared, the Byzantine had a foothold in the north-west of modern-day turkey, while the rest was occupied by populations of turkic descent. In 1453 Constantinoples falls to the Ottomans, and with it the former Eastern Roman Empire. The Ottomans dominated the Eastern mediterranean up until the First World War. To answer you question, the reason why the greek language is not as widespread as it should be is because the territories that were influenced have been under foreign rule (and by foreign I mean non-Hellenistic) for too much time.

I've learned too much from Reddit to assume that actual historical discourse is wanted. But y'all need:

Horrocks, G (2010) A History of the Greek Language and its Speakers. Cambridge

and

Kaldellis, A (2008) Hellenism in Byzantium. Oxford