For example, the Romans used the Colosseum as an entertainment place. At which point did they decide to " You know what? We will just abandon this building and not demolish it so people hundreds of years from now could visit it as a tourist destination".
An how did pagan temples (Ancient Egyptian, Hellenic and so) survive? Until this day we see zealots destroying places of worship like how ISIS for example did with many ancient churches and Shia mosques. I doubt the Catholic church for example was more tolerant than people nowadays, or was it?
Mostly because they were given another use.
For example, the pantheon in Rome was made into a Christian church and the Parthenon in Athens had various uses over the centuries, including as ammunition storage by the Ottoman Empire, which led to part of it exploding.
The Colloseum is an interesting one. It was left derelict for a very long time and I seem to remember it being filled with water at one point. The sculptor Cellini writing in the late 1500s describes it as a mystical place where people would go to summon the spirits of the dead. So it obviously had a special meaning to people.