I think you’ve confused the institution of the caliphate with the empire it controlled for part of that period. After the sack of Baghdad in 1256, which ended the Abbasid Empire, the Abbasid caliph resided in Cairo, where he had no political authority (the caliph was used to legitimize the rule of the Mamluk sultans, as he was their “guest”), and not much religious clout, if we’re being honest. The title of caliph was “gifted” to the Ottoman sultan Selim I in 1517 when he conquered Egypt under circumstances that are murky at best. So, for the entirety of that period, there was someone who had the title of “Abbasid caliph” but no actual authority.

However, even before the sack of Baghdad, the Abbasid empire hadn’t been the dominant political or military power in the Islamic world for quite some time. While the so-called “Golden Age” of Islamic civilizations is generally considered to have been during the early part of the caliphate (750-950ish), behind the scenes there was often political turmoil.

The Caliphs reigned as mere figureheads for at least four different groups of political or military factions brought in as advisors or aides, but wound up seizing as much power as they could and bending the caliphate to their will — this was the case repeatedly after about 900 AD or so. By 1000, there were three rival caliphates: the Umayyads of Spain, the Fatimids in Egypt, and the Abbasids in Baghdad. Although the Abbasids outlasted both dynasties, they lost territory to, among others, the Crusaders, Ayyubids, and Mameluks (Syria and Egypt), and the Seljuks (Iran and what is now Turkey) before the Mongols put an end to their empire.

If you want to read more, Hugh Kennedy is pretty much The Guy on all things Abbasid.