The first thing to note is that the East was never really Latin speaking at all. The lingua Franca of the East was Greek, but even that was most prevalent in the big cities like Antioch and Alexandria when you move south and east from Anatolia (Local languages dominated the countryside, where the majority of the population lived, e.g. Syriac in Syria, Demotic in Egypt to give the big examples, but lots of local languages and dialects were prevalent). Although Greek and Latin were widely known throughout the east, Greek was much more common throughout the Roman period and was probably more important in trade than Latin empire-wide. As far as Latin native speakers as Emperors, Justinian I (527-565) was the last Emperor to be born in a Latin speaking area (Tauresium, in Illyricum), so he is almost certainly the last Emperor who was a native speaker of Latin.

It seems like you’re mostly thinking administratively, where you’re exactly right that Latin was the primary language of official documentation and imperial proceedings. Justinian also features promisingly in this, as all of his major administrative writings (e.g. the Codex Iustinianus and the Digests) were first written and distributed in Latin. But by this point, not many individuals spoke Latin natively in the capital and most of the administration, though they were quite competent in it, were Greek speakers, and had a long tradition of lofty Greek literature to fall back upon while Latin was purely administrative. After Justinian, public documents were still written in Latin, but they were increasingly being published in Greek first until Latin fell completely by the wayside by the 8th c.

As for Roman numerals, Greek also has a numeral system, so when the shift to Greek language in administrative documents happened, so did the shift to Greek numerals.