The idea of governments acknowledging other governments as legal successors of past polities is a bit of a modern things. However, the Byzantine Empire was definitely seen as the successor of the old Roman Empire by many of its neighbors well after the Pope crowned Charlemagne. Really, though, it was less like they acknowledged the Byzantine Empire as the successor to Rome (like how the Russian Federation is recognized as the successor to the USSR in many treaties) and more like they just considered it the Roman Empire - in that they literally called it various versions of "Rome." The Arabs called it Rûm, which derived from the Greek Rhomaioi, meaning Romans. The Byzantines called themselves Romans, after all, so why would someone in the Arab world decide against calling the Romans "Romans" just because the Pope had created another political entity that was a rival claimant to the name "Roman Empire?" If you were an Arab, you probably wouldn't change your vocabulary for such a foreign reason. When they talked about the Crusades - which frequently involved forces from the Holy Roman Empire - the Byzantines are still called Romans whilst the Western Europeans were called by various other names (the most common being "Franks").

When Mehmed the Conqueror conquered Constantinople, therefore effectively ending the Byzantine Empire, he took the title of Kayser-i Rum - Caesar of the Romans - though later sultans disregarded that title.

Now, because the Byzantine Empire was very Greek and because it was known as Rûm, the term can also refer to Greek and/or particularly Eastern Orthodox (and some other versions of Eastern Christianity) communities, particularly those that lived under the Ottoman Empire. For example, the [millet](https://en.wikipedia.org/wiki/Millet_(Ottoman_Empire%29) created for the Eastern Orthodox subjects of the Ottoman Empire was called the "Millet-i Rum." So a term that derives from the Greek word for "Romans" eventually took on a whole new meaning due to the fact that the Arabs and Turks were quite willing to call the Byzantine Greeks "Romans."