Looking at medieval times, philosopher John of Jandun wrote his Treatise on the Praises of Paris in 1323 and singles out three buildings for praise: the Notre Dame Cathedral as you mentioned, the Conciergerie, and the Sainte-Chapelle. All three buildings are on the Ile de la Cite so I think this island in its total could be considered the icon of Medieval Paris

I think you could also make an argument for the Sorbonne as a major icon of Paris. Back in the Middle Ages I definitely think you could consider it the intellectual capital of Medieval Europe in terms of the thinkers it produced in this period.

I am not a historian, but I am fascinated by your question and have been reading about it all morning when I should have been doing other things. I will present a brief synthesis of my findings and my -- again, non-professional -- conclusion, which is...

TL;DR: There was no "icon of Paris" before the Eiffel Tower.

First and foremost, I recommend Roland Barthes' 1979 collection of essays The Eiffel Tower and Other Mythologies, which helped shape my thinking on this question quite a bit. The first essay in the collection is about the Tower. He begins by summing up the same position you take in your question:

The Tower is ... present to the entire world. First of all as a universal symbol of Paris, it is everywhere on the globe where Paris is to be stated as an image; from the Midwest to Australia, there is no journey to France which isn't made, somehow, in the Tower's name, no schoolbook, poster, or film about France which fails to propose it as the major sign of a people and of a place: it belongs to the universal language of travel (3-4).

Now, to sum up his argument about the Tower, allow me to quote this excerpt from the collection's first essay on the eponymous landmark:

This pure -- virtually empty -- sign [the Tower] -- is ineluctable, because it means everything. In order to negate the Eiffel Tower ... you must, like Maupassant, get up on it and, so to speak, identify yourself with it. Like man himself, who is the only one not to know his own glance, the Tower is the only blind point of the total optical system of which it is the center and Paris the circumference. But in this movement which seems to limit it, the Tower acquires a new power: an object when we look at it, becomes a lookout in its turn when we visit it, and now constitutes as an object, simultaneously extended and collected beneath it, that Paris which just now was looking at it (4).

So not only is the Tower, through its height, geographical placement, and aesthetic, constantly seen (indeed, Barthes notes that "you must take endless precautions, in Paris, not to see the Eiffel Tower; whatever the season, through mist and cloud, on overcast days or in sunshine, in rain -- wherever you are, whatever the landscape of roofs, domes, or branches separating you from it, the Tower is there" (3).) but it unifies all of Paris under its gaze. Moreover, this is its only real function. Built as the entrance to the 1889 Exposition Universalle in Paris, the Tower was quickly criticized as "useless and monstrous" in the famous Artists' Petition, published in the Parisian newspaper Le Temps. To ascend it, to look down on Paris from above, was to coordinate "the city into a kind of nature; it constitutes the swarming of men into a landscape, it adds to the frequently grim urban myth a romantic dimension, a harmony, a mitigation; by it, starting from it, the city joins up with the great natural themes which are offered to the curiosity of men: the ocean, the storm, the mountains, the snow the rivers" (Barthes 8) Barthes goes on to say:

To visit the Tower, then, is to enter into contact not with a historical Sacred, as is the case for the majority of monuments, but rather with a new nature, that of human space: the Tower is not a trace, a souvenir, in short a culture; but rather an immediate consumption of humanity made natural by that glance which transforms it into space" (8).

The Eiffel Tower is the icon of Paris, then, because it is Paris. That is its function -- perhaps not its original intention, but nonetheless the Tower (which "attracts meaning, the way a lightning rod attracts thunderbolts" (Barthes 9)) has become metonym and icon for the city.

I would contend that none of Paris' other famous structures or monuments do the same thing as the Tower, though the obvious contender is, as you suggest, Notre Dame. After all, in his liberetto Les mariés de la tour Eiffel, Jean Cocteau has his characters proclaim "The Eiffel Tower is a world, like Notre-Dame. It is the Notre-Dame of the Left Bank." "It is the Queen of Paris." It was started in the time of Charlemagne and, as /u/VaughanThrilliams (great name) said, it was picked out for its prominence and splendor pretty much as soon as it was built. Seeing it today, Notre-Dame is still spectacular, and it definitely is one of the icons of Paris -- but it is not, nor was it ever, the icon in the way the Tower is (and in the way your question situates it).

Prior to the construction of the Tower, I think that Paris was an icon in itself -- it was, as Walter Benjamin famously described it, "the Captial of the Nineteenth Century" -- the paragon of fashion, industry, arts, and culture of all kinds. Inasmuch as the Eiffel Tower, the largest structure in the world when it was first unveiled, the gateway to an international event designed to showcase art and human achievement, is Paris (an empty signifier, Barthes' lightning rod of meaning), it has entered into popular culture vernacular as shorthand for the city itself. Nothing before could have done that. Nothing before it was so thoroughly magnificent yet ambivalent, so thoroughly the place where it stood, as to warrant iconicity of the same magnitude.