The Islamic star and crescent is the symbol of Islam today

In order to regard something as Islamic, in this case the star and crescent symbol, it has to be regarded as such by a source of our knowledge of Islam, the Qur'an and/or Ḥadīth, or, at least, be endorsed by a figure, of some degree of perceived religious authority. The use of the star and crescent symbol by some Muslims, several centuries after Islam came to be, is not sufficient to warrant the characterization of it as an Islamic symbol; in reality, the star and crescent has no meaning in an Islamic context (I understand the common association of this particular symbol with Islam as a religion, but it's more a fallacy than a fact). It's also worth noting that the star and crescent symbol can be seen in use by entities with no Islamic affiliation, like, for example, the City of Portsmouth (in the UK).

What symbol was used to represent the symbol before the rise of the Ottoman Empire?

There were several states that appeared prior to the rise of the Ottomans with varying degrees of influence. I will focus on two ruled by the Arabs.

The Arabs opted for epigraphs of an Islamic nature rather than an actual symbol. Being calligraphs, they would be used as a state symbol, minted on coins and inscribed on state buildings. The Umayyads of Damascus and the Nasrids of Granada, are good examples.

Ziyad bin Abih, serving as governor of Kufa between 661-674, was the first to introduce a religious slogan, in Arabic, on his coin issues as representative of state identity; it read Bismi Allah Rabi ("In the Name of God, My Lord"). Coins, from that period onwards, started to be issued with similar phrases, and they differed from one governor, and region, to another.

Abd al-Malik bin Marwān, reigned between 685-705, introduced a uniform coin issue which stated, among other things, the Muslim declaration of faith: There is no god but God. Muhammad is the messenger of God. This coin issue would be continued, with minor changes, by the Abbasids and some Fatimids rulers (and would appear in the flag of modern-day Saudi Arabia).

The Nasrid rulers, in the Emirate of Granada, chose this, "La Ghalib illa Allah," which can be translated as "there is no victor but God," and used it for their flag and coat of arms, which appears at the Alhambra.