I realize that the city was retaken by "Byzantines" but wouldn't that just make it part of the Empire of Nicaea?
There's always more to be said, but in the meantime you may find some answers in this previous post by u/enygma9753 about the Byzantines' geopolitical situation in the years preceding the Ottoman siege of Constantinople of 1453, and the West's reluctance and inability to respond in time.
Both the 1204 capture and sacking of Constantinople by Latin Crusaders and the 1453 Ottoman siege were seen as a "fall" of Byzantium. The existence of rump successor states muddied the waters as to when the Byzantine Empire ended, with competing national interests laying claim to the Byzantine legacy. The 1453 siege marked its end both in a practical and existential sense for the West.
Constantinople was the bastion and capital of the eastern Roman Empire aka Byzantium or Byzantine Empire. For centuries after Rome had fallen to barbarians, it contained the knowledge of the ancient Greco-Roman world and priceless artifacts dating back to the time of Christ. As long as these remained in Christian hands, there was still hope in Christendom that Byzantium could survive. They could not make such a claim once it became the capital of the Ottoman Empire, with the great Hagia Sophia church converted into a mosque.
Political and religious differences fermented divisions between the Latin West and Orthodox East. The rise of Ottoman hegemony in the region forced both sides to try to reach a compromise, but their efforts to form a united front proved to be lacking by the 1450's. Its loss reverberated across Christendom, prompting fears of the imminent collapse of the West. Some even regarded its fall as a harbinger of the Apocalypse.