This is a highly difficult question to answer because the line where Rome "ends" and Byzantium "begins," is really up to the whim of whatever Historian is talking about it in whatever writing you're reading. Some say Justinian was the first Byzantine Emperor, while his Empire was much more similar to that of Trajan or Constantine than that of Basil and Alexius. Others say the first Byzantine Emperor was Arcadius, where his Empire was functionally (in organization and general spirit) the exact same as the Roman Empire it supposedly no longer was.

So, for the sake of ease of contrast, I'm going to define "Byzantine" as the Empire of Basil II, and "Eastern Roman Empire," as the Empire of Xeno.

Differences

1. Political organization. The Byzantines employed a provincial system called the Themata, or basically small military regions. Each was run by a Strategos who had both military and civil authority in their territory (although this changes a bit with the institution of judges, but I'll just skip that part). This was important because of the frequency of raiders, the local governors needed to be able to quickly get every resource at their disposal to be able to repel military incursions, and the unification of civil and military authority accomplished this. In contrast, the Eastern Roman Empire employed the traditional system of provinces, praetorian prefects, and magister militums, separating military and civil authority.

2)Religion. The Eastern Roman Empire, although still quite concerned with religious matters and very willing to get involved with it, such as Zeno's blunder that was the Henotikon, were generally much more lax with their religion. Religious debate was the new philosophical debate of the day, and various non-Chalcedonian sects would oftentimes enjoy passive tolerance or unofficial protection. Monophysites were generally allowed to do their thing, and only sparingly did Eastern Roman Emperors get involved in minor doctrinal disputes. In contrast, the Emperors of Byzantium were incredibly influential in the religion of the Empire. Whether it be the proclamation of iconoclasm under Leo III or Leo V, to things like the Macedonian emperors having what appears to be a fondness of killing Paulicians, the Byzantines put much more stress on the maintenance of orthodoxy in their Empire (whatever that meant to the Emperor).

3. Removal of all visages of republic. I'll just link my post here for that.

But really, other than that, I can not in good conscious say there are other "fundamental" differences. They considered themselves Romans, the Senate was still a body that the Emperor could turn to for legitimacy or otherwise ignore, the Emperor was an autocrat. The other differences are more so superficial, as is expected when switching languages as your language of administration, but that's really about it. The term Byzantine Empire is only a modern one used to help distinguish the classical Empire from the Christian one. You could rewrite a Byzantine History book and replace every usage of the word "Byzantine" with "Roman," and the book would still be 100% accurate. The Eastern Roman Empire simply adapted to the different political situation, it didn't fundamentally change itself in any incredibly significant way that made it any less "Roman."

There's no difference- fundamental or not. It's all in the terminology. There's some could answers on why we have the different names here.