Interesting question, and complicated. The exterior of the Pantheon has undergone a long and fraught history of neglect, repair, neglect, repair, alteration, and restoration since the end of antiquity. The building was completely abandoned by the end of the 6th century CE. It had flooded multiple times and its interior was probably stuffed with a meter or more of Tiber mud. The exterior suffered from lack of maintenance depending on the whims of various popes, and many of the original ancient Roman features of the exterior simply collapsed and fell off. The original tiling of the roof was taken away in one go, as was most of the bronze fitting from the porch (for fortification of St Angelo). This leads to the clever quip that the Barberini (the popes, e.g.) completed what the barbari could not. The structure itself, of course, is much too sturdy to collapse after a mere 2,000 years, including the dome. It is only the attached decoration which has fallen off or been removed.

It is a common misunderstanding that the interior is largely untouched. A study in the 1990s concluded that over 50% of what we see inside the Pantheon is either from the 17th century or later, or else is an ancient stone which has been put back (often requiring reshaping). Many of the original stones are gone and have been replaced in the intervening millennia with similar, but much less expensive variants. The porphyry, for instance, insanely prized and expensive, has been mostly robbed and replaced with common rosso antico or even with common grey granite. The abundance inside of giallo senese from Siena is a smoking gun of post-antique fiddling. Despite that, we are reasonably sure that most of the design of the interior flooring is preserved as far as shape and color. On the interior elevation, nearly all the original serpentine is gone, and replaced with the more pale and less expensive verde antico (most of which is probably ancient, but from elsewhere). For a complete rundown of the various meddling on this building, see Arnold Nesselwrath, Impressions of the Pantheon in the Renaissance, Cambridge 2015.