They kept their religious title under many Muslim dynasties who owned Baghdad and after the Mongol invasions in Cairo

The first part of this is true (with significant exceptions: e.g., the Spanish Umayyads, 929-1031, and the Almohads, 1147-1269); the second part, only for a limited geographical area. The 1258 Mongol sack of Baghdad was, for many Sunni Muslims (not just the Ottomans), the death knell of the ʿAbbasid caliphate. But even here, opinion was divided on who could lay claim to the position, or whether it was even necessary in the first place. Even before the fall of Baghdad, the political theorist ʿAbd al-Malik al-Juwayni suggested that a time might come when there would be no caliph; a relevant translation of parts of his argument is provided in the linked text, but I'll briefly quote here for convenience:

560. As far as the goods entrusted to the imams are concerned, if an era ends up lacking an imam and is without a strong, skilful and capable authority, such goods are entrusted to the ulama. Men, in their various classes, have a duty to refer to the ulama, who will express their opinion on all the issues of government. If they act in this way, they will be guided to the right path and the country’s ulama will become their rulers.

So, according to al-Juwayni, in the absence of a unifying monarch, the responsibility of rulership should devolve upon the community of religious scholars, essentially decentralizing the caliphal authority. (This is very similar to the Shiʿi theory of vilayat-i faqih, the basis for clerical rule in contemporary Iran in which religious scholars are presumed to stand in as authority figures for the occulted twelfth imam.)

...of course your question was about the Ottomans, who clearly did believe in the continuing relevance of a caliphal authority (at least, once they could claim said authority for themselves). But even here, the Ottoman claim to the caliphate was hardly unique; what makes them stand out is that they had the force to back it up, and the longevity to extend that claim over the course of centuries rather than the lifetime of a single ruler. In 1380, for example, the Anatolian physician Hacı Paşa used the title "khalifat rasul Allah fi al-muʾminin" to refer to Isa Beg, the ruler of the Aydınid beylik in Western Anatolia. (Admittedly, this was an exceptional case: even though political theorists throughout the [Sunni] Muslim world were divorcing community leadership from the rigorous requirement of Qurayshi descent, most authors avoided mentioning the specific title of caliph, seemingly happy to let the issue slip into obscurity. But it does go some way towards denying the Ottomans any claim of priority.) And just as Cordoban and Almohad claims to the caliphate in al-Andalus challenged the ʿAbbasid claims to universality in the medieval period, the Saʿdid ruler Ahmad al-Mansur (d. 1603) of Morocco challenged Ottoman claims in the early modern.*

This is not to mention a more mystical interpretation of the caliphate by Sufi groups, who appropriated it along with more narrowly temporal titles as a way of signalling the authority of eminent teachers: thus, even before Hacı Paşa's adoption of the title on behalf of Isa Beg, the Mevlevi poet and biographer Ahmet Eflâki (d. ~1359) referred to Rumi's successor Hüsameddin Çelebi as "khalifat al-Haqq" (al-Haqq, "the Truth," being another name for God).

It's worth pointing out an important distinction between the title used for Isa Beg and that used for Hüsameddin. The first, khalifat rasul Allah, denotes its bearer as successor to the Prophet Muhammad in temporal as well as spiritual authority and was traditionally used for the historical caliphate; the second, khalifat Allah [or al-Haqq or any of the other names of God] denotes a the bearer as God's viceregent on Earth and usually has more narrowly spiritual implications.

The mystical side of the caliphate is more relevant than it may seem: early Ottoman flirting with caliphal ambitions took place in the general context of the regime's post-Timur** recovery, a time when the Ottoman dynasty began to draw on Sufi traditions and terminology to justify its rule. By the early- to mid-fifteenth century, Murat II was being described as "halifetullah" [khalifat Allah]. The title became more heavily emphasized, however, in the reign of Süleyman I. Was this because, as a persistent historiographic legend suggests, the last Cairo caliph transferred the title to Süleyman's father Selim I on his conquest of Egypt—and perhaps more importantly, Mecca and Medina—in 1517? Because the Ottomans needed a way to counter their Habsburg rivals' universal ambitions? Did it represent any political ambition at all, or was it merely a continuation of the mystical caliphates of Hüsameddin Çelebi and Murat II? While the sixteenth-century claims to the caliphate are undoubtedly significant, there's a good nature of debate as to the nature of that significance: your position on this question will be a major influence in how you answer the question of why the Ottomans were seemingly unique in this respect.

If the new claim to the caliphate was a political claim to universal rulership, the Ottomans differed from their predecessors in just how possible it was for them to achieve such an ambition. (Universal rulership ambitions were also a legacy of the Mongols, although couched in rather different terms, so one might argue that the Ottoman state's particular relationship to Mongol theories of legitimacy made them more likely to usurp the caliphate than, say, the Ayyubids or Buyids.)

If, on the other hand, we see the claim to the caliphate as a continuation of the mystical tradition and an outgrowth of the Ottoman use of Sufi titulature, then it's the Ottoman relationship to theories of sacral kingship that's the limiting factor. True, the Ottomans weren't the only early modern Islamic state to operate on a theory of sacral kingship—but their chief rivals in this respect, the Safavids of Iran, were Shiʿi and therefore unlikely to advance any claim to what they saw as a corrupted institution. As I mentioned earlier, the Moroccan Saʿdids (whose claim to legitimacy was based in large part on their descent from Muhammad) did make a claim to the caliphate in the time of Ahmad al-Mansur; it just wasn't as enduring as the Ottoman claim.

This is not, of course, to downplay the importance of the Ottoman claim to the caliphate—which, unlike Ahmad al-Mansur's or Isa Beg's (or, for that matter, later "restorations" such as that of Usman dan Fodio or even ISIS), was recognized outside of the Ottoman domains. But again, this depended in large part on the state's ability to back those claims with force and enshrine them in diplomatic agreements (most famously in the 1774 Treaty of Küçük Kaynarca, where the Ottomans accepted Russian protection over Ottoman Orthodox Christians in exchange for recognition of its claims of protection over Russian Muslims). Even as the "sick man of Europe," the Ottoman Empire was one of the most powerful Muslim states in the world, and its longevity and prestige continued to serve as marks of divine favor for Muslims around the world.

* Ahmad al-Mansur used the caliphal titles imam and amir al-muʾminin far more often than khalifat, but on at least one occasion he explicitly referred to his realm as a "caliphate" in a letter to an African ruler (allegedly Askia Ishaq II of Songhay, but with some doubt—John Hunwick, for example, prefers Mai Idris Aloma of Bornu) chastising the recipient for his refusal to properly acknowledge Ahmad's claims: "you should obey his noble commands in regard to that complete obedience that God has imposed towards this Prophetic caliphate; and that you should attach yourselves, through an oath of allegiance to it, to the cord of the community..."

** Wikipedia says 1362, but this is a case where the Wikipedia article is probably just straight-up unreliable: the 1362 date has its origins in a sixteenth-century compilation of earlier chancery documents edited by a bureaucrat known for projecting the then-contemporary image of the sultanate onto the dynasty's early past.