There are many Muslims even today who argue how legitimate the Ottomans were as a Caliphate due to something about not being descended from Muhammad’s tribe.
How important was this in legitimacy of Caliphs? What did the Ottomans think? And what did Arab Muslims of the time think of it?
l'll try to answer your question - or at least the first two - sorry if it doesn't satisfy you. It's actually something that even jurists and scholars of the Ottoman era debated! Being a non-Arab dynasty, the Ottoman authority couldn't be legitimized in the form of a caliphate on the basis of the juristic canon. However, the Ottomans had ways to get around this, from alternative opinions, Sufi rhetoric, and factual legitimacy
So the Ottoman dynasty wasn't an Arab dynasty, nor do they claim kinship to Prophet Muhammad (though there are several unofficial attempts at that), so the dynasty legitimized their claims through alternative juristic opinions and factual legitimacy, that is a practical policy to arouse positive feelings of comfort and security amongst the people to propagate religiosity and traditionalism, to instill the image of a victorious Sultan, prosperous state and generous ruler. In Ottoman case, this was mostly done by ensuring the protection of hajj caravans and the security of the Hedjaz, which will we get through later. Let's examine some alternative opinions first. In 1544, Grand Vizier Lutfi Pasha, penned a treatise titled Khalas al-Umma fi Ma'rifat al-A’imma (The Salvation of the Islamic Community through Recognizing its Leaders) where he discussed the validity of the hadith saying that a caliph must be of Quraysh tribe. Lutfi argued that it's not valid that it was not acceptable to deduce a regulation from the hadith that people of non-Qurayshi descent could never become caliphs. He added that even if valid, it's only legally binding in the earliest times of Islam, during the times of the Rashidun Caliphs (632–661), since most independent Islamic rulers, including the the Mamluks, who were slaves and supposedly lower than the Ottoman Sultans claimed the title at this point. Lutfi further added that a ruler may become a Caliph via number of ways, including isti'la, or domination:
If the conditions mentioned above are combined in one person—to wit, conquest, power of compulsion, maintenance of the Faith with justice, command of the good and prohibition of evil, and the general leadership—then he is a sultan who has a just claim to the names of Imam and Khalifa and Wali and Amir, without contradiction
Additionally, Lutfi further demonstrated the legitimacy of the Ottoman ruler by arguing that his very Sultan title meant caliph, imam, and zill Allah (shadow of God) at the same time, so if a ruler managed to be the dominant power (the 'Sultan') he automatically entitled to those titles. Now Lutfi doesn't pull this out of his ass, this kind of alternative opinion had existed ever since the medieval era. For example, Hanbali jurist Ibn Taymiyya wrote in his work al-Siyasa al-Shari'a (Governance in Accordance with the Sharia) did not consider the rigid conditions elaborated in the classical theory as necessary for the establishment of the caliphate, especially the Qurayshi descent part. Many other Sunni scholars also wrote in the same vein as Ibn Taymiyya, like Hacı Paşa, Tarsusi, Al-Mawardi, and Ibrahim Muhammed, all opined that a Qurayshi descent was not a strict condition for someone to become a caliph.
Moving on, other than alternative juristic opinions, Sufi rhetoric were also used to justify Ottoman claim to the caliphate. Especially since the Ottoman dynasty was quite close with the Sufis and there are many Sufi proponents of the Ottoman Empire. They envisioned the caliph to be khalifat Allah (God's Deputy) and attributed to the Ottoman ruler the same spiritual qualities and powers according to the qut'b (axis mundi), the invisible perfect human being that God entrusts the management of His whole creation in Sufi cosmology. Ebu’l-Fazl Münşi, for example, stated that padishahi (rulership) was the khilafat-i khuda (viceroyalty of God), therefore equating ordinary rulership with the indirect rule of God. Using similar terms, Dizdar also pointed out that a sultanate is a caliphal vicegerency from God. In the same vein, Ensari opined that walaya (rulership) and sultanate together were equal to khilafat al-nubuwwa (Viceroyalty of the Earth). In contrast to alternative Sunni opinions stated by Lutfi, Ibn taymiyya and others that "reduce" the status/requirements of a caliphate so they can be claimed by ordinary sultanates, Sufi rhetoric elevated the status of a sultanate so they can claim the caliphate.
Now to factual legitimacy. There are many ways the Ottomans used to bolster their legitimacy via this method, but i'll talk the one that's most important for the topic. After the conquest of Egypt by Selim I, the Ottomans eagerly claimed the title Khadimu'l-Haramayn (Custodian of Two Holy Cities), to further solidify their claims as a Caliphate. Immediately following Selim’s conquest of Egypt, the Sharif of Hedjaz sent an official delegation to Cairo led by his own son to meet the sultan and, in a gesture heavy with political significance, to hand over to him the keys to Mecca. It was only after this public act that Selim, in a letter to the ruler of Shirvan, seems to have for the first time referred as the khilafet-i aliye (exalted caliphate) and to the insertion of his name in the hutbe of Mecca, reasoning that the failure of the Mamluks to protect the hajj routes had made it an obligation for him to assume that double role. Indeed, protecting the holy cities of Mecca and Medina and the routes leading to them is an important task undertaken by de facto Muslim power in the area ever since Saladin used the title to undermine Abbasid caliph's authority in Baghdad, because it meant that by keeping the holy cities of Islam and all kinds of Muslims heading towards it safe, Muslims would recognize them as the one rightful for the title of caliph. Ottoman claims to the title impressed many Muslims, including Muslims overseas to bolster their legitimacy.
Sources:
Legitimizing the Order: The Ottoman Rhetoric of State Power edited by Hasan Karateke and Maurus Reinkowski
Caliphate Redefined: The Mystical Turn in Ottoman Political Thought by Huseyin Yilmaz
Tordesillas and the Ottoman Caliphate: Early Modern Frontiers and the Renaissance of an Ancient Islamic Institution by Giancarlo Casale
This question is much younger than you might think. The usual claim is that Sultan Suleyman the Lawgiver (1520-1566) was the one to claim the title of caliph. However what was mentioned in his father's letters (Sultan Selim the Grim, 1512-1520) was not a title of caliph, but of caliphate. Although some sultans may have used the title, and although there may be a couple of Arabic commentaries on this Ottoman practice, in reality little significance was given to it: you might see evidence of it in even Sultan Mahmud II (1808-1839) sharing this title with Fath Ali Shah, ruler of Iran, in the Treaty of Erzurum (1823).
This changed with Sultan Abdulhamid II (1876-1909). He proclaimed himself caliph and the protector of the religion of Islam. He wasn't recognized by Sultan Mawlay Hassan of Morocco, Khedive Ismail (in Egypt) or Nasir ad-Din Shah of Iran. However, those Muslims that lived under the colonial regimes in Asia and Africa did see him as rightful caliph, although the primary reason was their belief that the Ottoman Empire was capable of defending them against European powers.
As for the Arab Muslims in the Ottoman Empire (save for Egypt), they did recognize the role of Ottoman sultans as defenders of Islam, but they were unwilling to recognize them as true caliphs.^(1) That resistance diminished thanks to European powers conquering one Muslim region after another, but even then they were quick to add that the title of the Ottoman caliphs was not the same as that of the Four Rightly Guided Caliphs. Some scholars, subduing to the propaganda emanating from Istanbul, accepted the stance that the Ottoman claim to caliphate was indeed legitimate.^(2) Of course, the anti-Ottoman line was not insignificant among Arabs.^(3)
Anti-Ottoman works in the Ottoman lands hardly proved to be an issue, as the Ottoman censorship was quick to remove anything tangentially dangerous to the regime. However, the Arabs in exile were not really concerned with Sultan Abdulhamid II's claim to caliphate: they were far more concerned with the restoration of the Constitution of 1876. The issue of caliphate was simply exaggerated by European imperial planners (as it would prove during the World War I).
_________________________________________________________
^(1) This reasoning emanates from Muhammad Amin ibn Abidin from Damascus (d. 1836), who wrote about the Ottoman claim of caliphate, justifying the need to rebel against Ibrahim-pasha. He made a clear line between the greater Caliphate of Four Rightly Guided Caliphs and the lesser Caliphate of the Ottomans.
^(2) For example, Abd ar-Razzaq al Bitar from Damascus (1837-1917) in his biographical dictionary ascribes the titles of Shadow of God on Earth and Commander of the Faithful to Ottoman sultans. These two titles were traditionally ascribed to caliphs alone. I'd also emphasize that he denied claims of Muhammad Ahmad of being the mahdi (Muhammad Ahmad led a rebellion against the khedive, and therefore against Sultan Abdulhamid II as well), due to the prophecy that the House of Osman would hold sultanate until the coming of mahdi (the prophecy had been written by Ibn al-Arabi).
The biggest proponent of this stance was Abu al-Huda as-Sayyadi (1850?-1909). This man from a village in northern Syria managed to become the head of the prestigious (if purely ceremonial) office of naqib al-ashraf (Head of the Prophet's descendants). This man went to Istanbul in 1878 and presented to the sultan, saying that he had dreamt of Prophet, who gave him some private information for sultan. However, as-Sayyadi spoke only Arabic, which sultan did not speak. Therefore he was dismissed, but he returned after two days, speaking fluent Turkish. After the sultan's spies confirmed that as-Sayyadi spoke no Turkish before coming to Istanbul, Sultan Abdulhamid II was amazed and took as-Sayyadi as the spiritual adviser. As-Sayyadi had great influence on the sultan.
^(3) Good example being Abd ar-Rahman al-Kawakibi from Aleppo (d. 1909). Having multiple run-ins with the government concerning his attempts to found Aleppo's first private newspaper; chafing under the regime, he moved to Cairo. Arguing that the Turks allowed Islam to decay, he suggested that only a renewed Arab Caliphate, with its capital in Mecca, could preserve the faith and keep Muslim lands free. It's interesting that one of al-Kawakibi's sources was The Future of Islam by Wilfred Blunt, a former Foreign Office employee. The British were most concerned about Sultan Abdulhamid II's claim to the caliphate, as it provided a great opportunity to destabilize their shadow regime in Egypt. Therefore the British agents managed to distribute and smuggle copies of al-Kawakibi's works in Ottoman Arab lands.