The Ottoman Empire is not considered a continuation of the Roman Empire because it wasn't a continuation. It can be argued with some amount of success that the Ottomans are one of the Successors of Rome, in the same way that the Goths (Visi and Ostro), the Franks (later the Holy Roman Empire) and even the Catholic Church are successors of the Roman legacy. However, stating that the Ottomans are a continuation of the Roman Empire in the same way the Eastern Roman Empire (what we now call Byzantine) is a continuation of the Roman Empire is simply not accurate.

You mention a few definitions for an empire: religion, language, territory, culture, and administration. Under closer inspection we can see all of these factors help demonstrate the continuity between Rome and Byzantium, while accomplishing the opposite for Rome and the Ottomans.

Let's start with religion. Religion is a rather poor defining characterization of a people or group of people in general. This is for a variety of reasons, but primarily because this can be chosen and changed by individuals. If a citizen of Roman Antioch in the 3rd century AD decided to join a cult devoted to the Egyptian sun god, Ra, he could do so. This didn't make him any less Roman, and there was a huge variety of religions active within the empire. This was actually one of the most successful policies of the Roman government, when new people were brought into the empire, their gods were either combined with the Roman versions (the Egyptian sun god Ra, might get combined with the Roman sun god, Sol) or just added to the pantheon along with the other gods. On the other hand, by almost all definitions the Emperor Constantine was a Roman Emperor but yet he legalized Christianity and in fact gave it legal advantages within the empire. Does this make Constantine non-Roman? I don't know of any mainstream historian who would make this claim. So again, Christianity is really poor evidence that the Medieval Roman or Byzantine state was any less Roman and in fact only strengthens the continuity between the late antique and medieval versions of the empire.

Next is Language, I'm assuming you are referring to Greek. The fact of the matter is that after the conquests of Alexander, along with Greek colonization, much of the Mediterranean world spoke Greek. In fact, most educated Romans spoke Greek during the Roman Republic, and to take the point even further, so many Republican Romans spoke Greek and were so heavily influenced by Greek culture that there was even a conservative backlash led by Cato the Elder against the Hellenization of the Roman state. The people who lived in the geographic regions of the Eastern Roman Empire before Roman conquest primarily spoke Greek, they continued speaking Greek during the Roman Republic, and didn't stop speaking Greek after their integration into the Roman state. If your point is that the Greek language is somehow un-Roman that simply is not the case. The Roman ruling class, as well as its people, widely used Greek though most of Roman history.

Next on your list is territory. As people can move and migrate this is also a pretty poor definition of a group of people. For example, when the Western Goths or Visigoths settled in Roman territory on the invitation of the Emperor Valens in the 4th century, did that make them Roman? Certainly not. When the Song dynasty lost their ancestral homelands, the original Chinese territories occupied by the Shang and Zhou dynasties, and were pushed into territory conquered later on, did that make them less Chinese? Definitely not. The same is true for the Romans, when Diocletian created the tetrarchy in in late 3rd century AD, none of the tetrarchs picked Rome as their capital. As time went on, the ancestral Roman or Italian heartlands became less and less important to the political, economic and military institutions of the Empire. Despite this, I don't know of anyone who would seriously argue that Diocletian wasn't a Roman Emperor.

Finally let's talk about administration. Administration is a more useful topic when trying to classify a people or state but again has some serious flaws, namely that it can change. Just to name a few obvious examples, the Roman Republic came to an effective end with the ascension of Augustus in the first century BC. Despite the rapid evolution from a republican styled oligarchy to a military-backed dictatorship, we call both administrations Roman. When Diocletian rapidly expanded the administration, increasing the number of officials in the Empire from a few hundred to tens of thousands, or when he divided power between four rulers instead of one, we still call it the Roman Empire.

When we try to describe different groups of people, states, civilizations, whatever you may want to call it, all of these above factors are valuable, but there are several points you're missing, firstly, political or authoritative continuity. Who was in power? And how did they get their power? When we talk about the Roman state, from the rule of the seven kings in ancient Rome, through the Republic, the Principate, the Dominate, and into the medieval Byzantine period, there is a continuity in rule. We can trace who was in power, from kings, to consuls, to princeps, to Emperors, all the way from Ancient Rome in 500 BC through Constantinople on May 29, 1453. If the Byzantine Empire really wasn't Roman, who was the first Byzantine Emperor? Whoever one might chose it would be completely arbitrary. For example, some consider Constantine a Roman Emperor and simultaneously the first Byzantine Emperor, he can be both because these two empires are one and the same, just evolving over time. If you were to put Constantine in Constantinople some 600 years in the future in the year 1000 AD, it would feel very familiar to him. Familiar language, religion, architecture, and government. However, if you were to put Constantine in Republican Rome, 400 years earlier, it would feel completely foreign. The government, the religion and customs, even the way people dressed, would all be drastically different even though there is a shorter period of time that has elapsed. The point I'm trying to make is that the Byzantine state evolved very slowly over a long period of time, in the same way that all peoples evolve and change. Yes it seems so obviously different if you take Byzantium in 1453 AD and compare it to the Republic in 500 BC, but there's a lot of time in the middle where the differences weren't so clear.

To contrast this evolution of a long period of time, let's talk about the Ottomans. On May 28th, 1453 there was an Emperor, in the same city, sharing the same name, religion, language, and empire as Constantine the Great, who lived over 1000 years in the past. Despite the vast amount of time that transpired, and all the had changed, we can trace the transfer of power from Emperor to Emperor over the entire span of the millennium. On May 30th, 1453, Constantine was dead, the entire much of the administration and ruling class was either killed or fled west, and replaced with a people that came from the distant Eurasian steppe, a territory never touched by Roman conquest. These new conquerors brought a new religion, and language that had never before been seen in the empire and did not continue Roman cultural and religious institutions but ended or destroyed them. This would be a completely different discussion if the Ottomans had taken a similar approach to the Jurchens or Mongols after conquering parts of China. After their conquests, the Jurchen and Mongols largely adopted Chinese culture, religion, language, dress and government. Because of this there was much continuity between the preceding and succeeding dynasties and over time these foreign conquered were almost indistinguishable from the Chinese they had conquered. Contrast this to the Ottomans, who brought in language, customs, religion, and governmental system and replaced the existing Roman institutions. My final point is that only looks at things like religion and language are simply not enough to define a people, these things are always changing and always evolving. What's important is why are these factors changing? Is it a natural evolution over one thousand years? Or something imposed by a foreign conqueror overnight? We have to be extremely careful to review all the data we have available, and not rely on a simple check list of differences or similarities.

Sources

J. N. Adams, Bilingualism and the Latin Language

John W. Chaffee, The Thorny Gates of Learning in Sung China

Elizabeth Endicott-West, Mongolian Rule in China

Treadgold, A History of the Byzantine State and Society

Alexander A. Vasiliev, History of the Byzantine Empire

edits for grammar and spelling