Well the caliphate was just about dead in the 13/14th century. No less a chronicler than Ibn Khaldūn once expressed that he felt as if he was writing Islam's obituary.
Its widely believed that the Islamic golden age ended in the beginning of 1258, when Hulaga Khan lay siege to Baghdad. Spanish contemporaries were launching pogroms and purging Islam from the Iberian peninsula in the Reconquista, and the crusade were on-going. On top of all of this the Islamic world was very dynastic, then as it is now, and there wasn’t really a caliph. Many Muslims considered the Abbassid Al-Musta'sim to be the caliph, but he was trampled to death by the horde. It really did look as if Islam might be done.

But there were the Turks... The Turks were a people with very similar ancestry to the mongols, they were originally nomadic horseman of the steppe who had migrated from the valleys of the Atlai mountains.
Unlike the majority of Mongols, they had settled in Asia Minor/ Anatolia/ Turkey. Also unlike many Mongols, they embraced Islam, which was good because the nigh unlimited supply of Muslim refugees fleeing the Mongols and the Crusaders were willing to rally under the Turks. The Turk in particular was Osman I or Othman (from whom, the Ottomans get there name). Ibn Khaldūn saw the coming of the Turks into Islam as proof of God’s continuing concern for the welfare of islam. These nomads, were, he thought, sent to revive the dying breath of Islam. The dynasty he Osman founded was a weird and fascinating amalgamation of Steppe, Western, Byzantine, and Islamic civilization.

In 1463, they conquered Byzantium/Constantinople/Istanbul. It was a sacking reminiscent of the appalling siege on Baghdad. Prior to the conquest, the Hagia Sophia had been the worlds largest Christian cathedral for almost a thousand years. During the siege, women, the young, and the infirm sought refuge there. By the end, it was a mosque, converted, not long after the mutilated remnants of rape victims had been discarded.

The Ottomans were feared and respected for good reason, they were notorious slavers, pirates, and conquerers. Perhaps the most famous and loved of all Ottoman Sultans/Caliphs was Suleiman the Magnificent(1520-1577). His prodigious stewardship saw the empire expand from Budapest, to Baghdad, To Tripoli. The Ottoman empire, was from thereon stuck between something of a rock and a hard-place. They needed to westernize to keep up with the other European powers, but if they did so, the Muslim populous would reject it. The Sultans had some strange ways of trying to pacify their people as well as their enemies. The most modern and elite Ottoman troops - the Janissaries, were all captured infidels. Sultans would tax their Balkan Christian population by taking the strongest son from each family as a slave. This served to keep the non-Muslim population weak, and the army strong.

Of-course not all slaves were soldiers, many were servants, laborers, even government officials. Some slaves were castrated to ensure they would not have children, this was particularly true of those imported from Africa.

The harems are of-course an interesting part of Ottoman society. The Imperial Harem was essentially a group of infidel women held against their will, guarded by African Eunuchs - also slaves. Africans were thought to be particularly strong and obedient, so were used to protect the facility. White slaves, were more likely to serve in the army, and were also castrated but kept their testicle, this is believed to be so that they kept producing testosterone.

When a concubine would produce a son, she was given elevated status becoming a Valide Sultan (Queen Mother). Interestingly, it was the etiquette for a Sultan to no longer mate with this woman, this was thought to be best as it ensured that the Son of a Sultan would have a mother all to himself, and this would beneficial to his up-bringing. This wasn't always the case however, Suleiman for example - famously fell in love and remained faithful to a Christain slave from the Ukraine named Roxelana.

The decline of the empire was long and slow, never really keeping up with the West, never really for-filling the desires of the Muslim populous. The nails in the Ottoman coffin were of-course the British, French, and quasi-allied Arab Bedouins in the early 20th century. T.E. Lawrence, or Lawrence of Arabia has a wonderful and fascinating take on all of this which he wrote about extensively in later life.

TL;DR - Ibn Khaldūn and T.E. Lawrence are obvious choices, but i think they are the right ones. They lived the beginning and the end respectively and both write about the Ottomans from the position of informed outsiders.