How would a building like The Pantheon in Rome burn down if it was made of stone and marble?

I don't have specific knowledge of the building techniques of the Pantheon, but this kind of question comes up pretty commonly regarding how medieval stone or brick castles and churches burned down. The answer is that even though the stone or bricks themselves are generally unaffected by the heat, the mortar laid down between the blocks IS affected, as the mortars chemically bound water molecules escape and the weakened mortar loses its strength, crumbling away or getting crushed by the weight of the wall above it. Load-bearing walls may collapse completely, while other walls may stay standing but no longer be structurally sound.