This is a somewhat tricky question to answer since there are several parts that underlie it. Since my knowledge is primarily on the early medieval period I will talk mostly of that.

First: The founding of the Empire is conducted by the Pope, which was a political move in order for the Pope to retain some independence from Charlemagne/Franks. At the time of the coronation (Christmas day 800) the Papacy was more or less in an alliance with the Franks. This began in 751 when the Pope anointed Pepin III as King of the Franks (replacing the old Merovingian dynasty). So at the time of the empire's founding, although it was not referred to as the Holy Roman Empire yet, Pope Leo had managed to maintain some sense of independence from the Franks.

Second: The so called "Donation of Constantine" was an important document that was 'discovered' in the mid-8th century which more or less claimed to give the Papacy claims to land and independent authority. I put discovered in quotations because it was a forgery, although this was not discovered until the 15th century.

Third: Finally, the early Holy Roman Emperors did in fact exercise control over the Papacy. For instance Charlemagne goes to Rome several times in order to either reinstate the Pope (as he does immediately preceding his coronation when he restores Pope Leo) or to quell disputes. Emperors such as Otto I also marched on Rome in order to exercise control over the city.

So essentially although the Empire did not control the city in terms of it being a part of the empire's territory there were reasons that Rome/the Papal States remained semi-autonomous but the Emperors did have some control over the Pope and the city, at least as far as my knowledge goes so through the 12th century or so.

Hopefully this helps!

Sources: Robert Folz "The Coronation of Charlemagne 25 December 800" Pierre Riche "The Carolingians: A Family That Forged Europe" Paul Collins "The Birth of the West" Joesph M. Woods 'The Rise of the Papal States up to Charlemagne's Coronation"

To add to /u/Quixxeemoto, the "true" remnant of the Roman Empire, the Byzantine Empire, was being ruled by a woman, Empress Irene, at the time of Charlemagne. There were also growing differences in Western and Eastern religious traditions that formulated into an argument that the position of Roman Emperor was vacant and occupied by a woman advised by heretics. After Charlemagne was crowned Holy Roman Emperor, he then offered to marry Empress Irene and fix the East's religious problems.

It's worth keeping in mind that despite it's name, the majority of the territory and economic/military power of the HRE was located in what is today Germany, Austria, the Low Countries, and bits and pecies of of a few surrouding nations, like Bohemia (today part of the Czech Republic) and Alsace (today part of France). The Emperors of the HRE ruled over a very decentralized state. Even during the Empire's "heyday" in the 13th century, the emperors often struggled to effectively project their power south of the Alps. The investiture controversy is in some ways a reflection of the major power centers in Germany and Italy each trying to influence the political affairs of the Rhenish bishoprics.

http://en.wikipedia.org/wiki/Holy_Roman_Empire#mediaviewer/File:HRR.gif

I enjoy this representation of the Holy romans empire's borders over time.