This is not precisely my topic - I study early modern Islamic societies - but you are right in that the star and crescent is a modern symbol, derived out of the heraldry of the nineteenth-century Ottoman Empire and copied, as that empire fell but the idea of its vanished caliphate gained a certain cachet, into the flags of other states.

There was no simple graphic "Islamic symbol" before that time. Islamic thinkers had always been wary of an image that represents religious belief (it had been a trope in Islamic medieval anti-Christian arguments that Christians "worshipped the cross"), so none developed. For an image to represent God or belief in God was dangerously close to idol-worship.

However, there were certain visual cues that unmistakably evoked Islam. One was writing in general in calligraphic Arabic script, which developed into a sophisticated art and was found all over buildings and flags and domestic objects and usually identified its bearer as Muslim (of course there are exceptions). Another was the simple and recognizable shape of the calligraphic "Allah" (الله) or "Bismi'llahi'r-rahmani'r-rahim" or even the declaration of faith ("There is no god but God and Muhammad is his messenger" لا اله الا الله محمد رسول الله) which represented monotheism even to an illiterate. Often in mosques you would see the graphically striking names of God, Muhammad, and his companions (الله - محمد - ابو بكر - عمر - عثمان - علي - حسن - حسين) or a Qur'anic verse or two with an identifiable visual form. This is as close as you will get to an answer.

There is also something called the "liwa al-hamd" (the flag of praise) that was allegedly carried by Muhammad in some battle and will return at the end of time to be held by Jesus or the Mahdi to lead the faithful to victory. It's mythical so descriptions of it differ, but a typical representation looks like this; other descriptions of it have the declaration of faith on it, which is part of why you see that on the Saudi (and ISIS) flag.

The Iranian Revolution used the tulip as a symbol. Modern Shi'i often use a forked sword (the Zu'l-fiqr carried by 'Ali) as an identifying image.

This is not to say that states identified with Islam did not have flags with symbols on them throughout medieval history. Especially notable for this is Mamluk Egypt, which learned a complicated heraldry from Crusading Europeans. Turkish tribes each had runic symbols that were preserved in Rashidu'd-din's History. EDIT: There's also the tuğra of the Ottoman sultans, a kind of imperial signature-stamp for documents (other states prior to the Ottomans also used tuğras, like the Akkoyunlu). It didn't have a religious connotation then, but it does now among a certain class of Islamist Turks who subscribe to neo-Ottomanist fantasies.