In both cases, the Pantheon was damaged but not entirely burned down. Further, the extremely durable building we see today is not an exact copy of the earlier Pantheon(s) - its design was largely a response to the damage caused by the earlier disasters.