All through Late Antiquity, although the city of Rome was much diminished in size and practical importance, it nonetheless retained enormous political and symbolic value. u/shlin28 wrote the be-all and end all on the specifics of what was happening in the city of Rome after the end of the Empire. You can thusly imagine that the early Bishops of Rome derived much of their authority as a consequence of their history and symbolism tied to their city, as indeed u/thejukeboxhero explains in this extensive discussion.

You will see in the second linked answer, however, that Rome's primacy was far from a foregone conclusion. It is only after the Carolingian conquest of Italy, when Frankish Kings looked to Rome for spiritual guidance and (more importantly) legitimization, that the practicalities of Roman supremacy over the church are ingrained.

And even then, as the Carolingian successor-states fragmented into what would become France, Germany, and Italy, the lines dividing religious and secular authority were very blurry; especially in Italy, where the church had appropriated the institutions, both physical and theoretical, which had characterized Roman law and order. While this might have strengthened the Papacy's authority, in the long run the Pope very well might have been drowned out by the multitude of competing Italian bishops. The Papacy's continued prominence can in a first moment be attributed to its usefulness in legitimizing the Ottonian Empire's tenth-century union of the German and Italian monarchies, and in a second moment in legitimizing the opposition to that union. I actually wrote about that dynamic, through the lens of the Pope's acquisition of temporal authority, in this somewhat cursory answer. And, as you will see in that answer, even though these very different forces all seemed to strengthen the seat of Rome, in the thirteenth century the King of France was nonetheless able to move the Papacy to the French town of Avignon for a period; again reinforcing the not entirely unassailable primacy of Rome as the center of the Catholic Church.

There seems to be a bit of confusion in your post though. The Vatican City is a fairly recent polity; for much of the Papacy's history, Rome was located in the aforementioned polity called the Papal States. A now-deleted user and I participated in this discussion about some elements of life in the later centuries of the Papal States. I wrote a short note about the foundation of the Vatican City, and how it is different from the Papal States, in this answer.