For over four hundred years the Emperor held the title of Sultan and Caliph. When the final Sultan Mehmed VI was deposed, his cousin Abdulmejid II was elected Caliph a few weeks later. If both positions were traditionally held by the same man, why did the Turks allow the Caliph position to still exist after removing the Sultan position? Obviously the Caliph was deposed do to Kemal's secularization of the country but it just seems odd that the new nation would still want an Osman in a prestigious office.
I don't know much about the later Ottoman empire, but the Turkish Caliphate wasn't taken particularly seriously even by the Turkish ulema at any point in it's history, at least as far as religious legitimacy was concerned. So there wasn't much danger in it using it's pretty much nonexistant religious authority to cause trouble. At the same time many muslims, not just in Turkey, liked the idea of there being a Caliph, even if they didn't necessarily want him to have control over... anything. The brief period where the Republic and Caliphate coexisted reduced it to a powerless mascot, even more so than the modern British Crown. He was finally banished mostly because Ataturk found him obnoxious, and a potential pawn to be used by reactionary forces. The Caliph himself was mostly concerned with painting and pinning bugs to pieces of wood.