Charlemagne's coronation is an interesting one.

To set the scene with relevant information: the title of Imperator of the Western Roman Empire had been vacant since 476, with the deposition of the child Emperor Romulus Augustulus by the general Odoacer. He chose to reign only as King of Italy and thereby renounced his claims to rule over the rest of the Western Roman Empire, which by that point had de facto fragmented into a series of successor kingdoms anyway. One of these kingdoms was the Kingdom of the Franks, who were a pretty heavily Romanized Germanic people, who originated in what is now the Low Countries, and who ruled over a mixed population of Gallo-Romans and non-Roman Germans straddling the Roman limes in what used to be northern Gaul. It's important to conceive of them as a fusion culture, which combined Germanic traditions of kingship and inheritance with strongly Roman civic traditions and forms: I think it's a fairly safe bet to say that they considered themselves to be Roman, and that their leaders consciously appealed to Roman public ideology throughout the post-Roman era (eg the Merovingian king Childeric I's funerary ring depicts him in a Roman military cloak, with long Germanic hair and moustache).

So, the idea of being Roman was certainly not an alien concept to the Franks, and certainly gives reason for why Charlemagne might select projecting himself as a Roman Emperor as a convincing source of legitimacy. And this was something which Charlemagne needed every shred of: he found himself in possession of a vast personal empire, created by campaigns in northern Italy, the Basque country and northern Spain and Saxony. He likely felt that such a vast territory couldn't simply be justified on force of arms alone; it required a unifying ideological claim for it to endure.

The next part of the puzzle is Papal politics. In Late Antiquity, the Bishop of Rome emerged as a strong secular ruler of the city and its environs after the Imperial capital decamped elsewhere (eventually settling in Ravenna). However, as the Empire fell, the Ostrogoths and Lombards established a permanent presence in Italy, and the Popes were continually in danger of coming under the domination of these 'barbarians'. Charlemagne's campaigns against the Lombards in northern Italy in the 770s demonstrated that he was an emerging European hegemon, potentially capable of securing Papal autonomy. Indeed, the coronation of Charlemagne as Emperor was not without precedent; we have letters to the Pope sent after the Lombard campaigns in which Charlemagne requested that in future the Pope pray only for him! The title of Imperator was one which the Pope reserved as one which he could grant personally, and the method of this bestowal was all-important: by granting the title of Imperator Romanorum ('King of the Romans'), the Pope would claim superiority over the Emperor - the gift glorifies the giver rather than the receiver. Nobody in the Early Middle Ages would have missed this dynamic, and it set the scene for several centuries of conflict between Pope and Emperor over whom was subordinate.

Thus, on Christmas Day 800 CE, Charlemagne was offered the Imperial crown by Pope Leo III upon a visit to Rome. The historical sources, such as Einhard's contemporary Vita Karoli, have Charlemagne totally surprised by the offer, and modestly refusing it before reluctantly accepting. But this is doubtless a piece of political theatre required by Christian mores - all of the puzzle pieces had fallen into place: the enduring Roman traditions of the Franks, Charlemagne's need for a unifying, legitimising ideology which would secure his empire, and the temporal realpolitik of Papal politics on the Italian peninsula.

To address your questions:

Your questions get at the heart of how we periodize history somewhat arbitrarily, especially in the fluid Early Middle Ages - saying 'this was the Carolingian Empire and that was the Holy Roman Empire' is largely arbitrary, and even the terms we use are convenient back-formations that would have been meaningless to those at the time ('Holy Roman Empire' is a term from the Early Modern Era, for example). I suspect that most Franks would have thought of themselves as Franks first and as Romans more generally, especially with the refoundation of explicitly Roman ideology under Charlemagne.

To extend the story of the Holy Roman Empire a little - the Carolingian phase of the Holy Roman Empire did not survive Charlemagne's grandsons, amongst whom the Empire was partitioned and dissolved into civil conflict. The title of Imperator, however, did not drop out of use: the Pope's found it to be a useful tool to secure their autonomy & prestige, awarding it to a succession of lesser German and Italian nobles who would best secure their interests amid the waning of Carolingian power, before the HRE proper was refounded by Otto I in 962 CE and his Ottonian successors. So whilst Charlemagne did not succeed in creating an enduring Imperial state, we can conceive of the experiment as more of another expression of the lingering common ideology of post-Roman Central Europe - for even when Frankia was partitioned and disunited (both before and after Charlemagne) there is considerable evidence of a sort of pan-European identity that permitted the continual re-emergence of large overlordships.