It's always much harder to answer the question of "why not" rather than "why," but I think it would be good to start by addressing the implications of your question, since it seems to be based on a questionable assumption.

What I mean by that is why weren't the Islamic world opposed to the clergy and the system of caliphate, and why didn't they want to modernize the religion?

The direction of developments was in fact the opposite, the Ottomans were in the process of developing a more rigorously articulated system of organization for the Sunni Islamic religio-judicial hierarchy in the empire, in comparison with the quite decentralized system which existed during the Middle Ages. But leaving that aside, what makes you think that "the Islamic world" should have been opposed to this? The Protestant Reformation was, after all, a response to conditions specific to Catholic Europe, which was not at all similar to the conditions prevailing in the Ottoman Empire. The notion of this being part of an effort to "modernize the religion" is also problematic, as it would seem to imply that Protestantism is somehow more "modern" than Catholicism. I don't think that the term "modern" is at all helpful here in understanding the conditions of the sixteenth and seventeenth centuries, and it is certainly not how contemporaries conceived of these issues. Rather, religious reformers tended to look to the past, viewing themselves as the restorers of a more pure and ancient version of their religion. Nor do I see how the strength of the Ottomans is at all related to their likelihood of experiencing a Protestant Reformation-equivalent.

All-in-all, asking why the Ottomans did not have a Protestant Reformation is a bit like asking why Medieval Europe did not ever experience a wave of Neo-Confucianism, or share in Islam's burst of Sufi mystic activity during the fourteenth century. The answer can only be that they were totally different places and that the circumstances were not directly comparable.

Now, that being said - it's not as though there were no religious reformers to be found in the Ottoman Empire during this time period. The most famous were the movement of the Kadızadelis, religious fundamentalists who based their thought on the teachings of the Ottoman scholar Mehmed Birgivi Efendi (d. 1573), particularly those outlined in his book at-Tarîkatü'l-Muhammedîye, or "The Way of Muhammad." Their goal was to purge Islam of all its "innovations" acquired since the time of the Prophet Muhammad, particularly those activities associated with mystical Sufi worship. They were particularly active in Istanbul and mobilized the population through their charismatic mosque preachers, and twice attained high regard with the Ottoman government, first under Murad IV (r. 1623-1640) and then under Mehmed IV (r. 1648-1687). However, they were regarded as fanatical and barbaric by several of the Ottoman Empire's leading intellectuals and mainstream religious leaders, who at that time were very tolerant toward Sufi worship - the fact that these same intellectuals were the ones writing the history books means modern historians have been given a quite negative view of the Kadızadeli movement. The movement maintained its influence until the Ottoman defeat in the 1683 Siege of Vienna, which had been highly encouraged by Kadızadeli leaders. After that point Mehmed IV decided to withdraw his support from the movement and banish its leaders from court.

On the Kadızadelis, important works include Madeline Zilfi's 1986 article "The Kadizadelis: Discordant Revivalism in Seventeenth-Century Istanbul" and Marc David Baer's 2008 Honored by the Glory of Islam: Conversion and Conquest in Ottoman Europe.