This is a very interesting question that will be a bit difficult to answer without going on a long case by case basis. But since nobody has tried to answer this question yet, I'll throw my hat in the ring.

Before I get to the Colosseum and Circus Maximus specifically I feel it's necessary I cover the relevant history of the city of Rome during Late Antiquity and the Early Middle Ages. As you may know Rome was during the 5th Century subject to two sacks, the first by the Visigoths in 410 A.D and the second one by the vandals in 455 A.D.

Contrary to popular belief however there's not much evidence of significant, irreperable structural damage to the core of the city. We know the Basilica Julia and Basilica Aemilia were burned during the sack, but outside of that the Visigoths appeared to have largely just stripped buildings of decoration and revetment. These were Romanized mercenaries after all, not savage barbarians. They weren't out for wanton destruction, but rather plunder and gold, of which Rome had plenty.

The Vandal sack in 455, due to it lasting longer is generally thought to have been more throrough. But once again the Vandals appear to have mostly stripped buildings, not destroyed them entirely. For example, the Roman historian Procopius stated in the 6th Century that the Vandals, during their sack had stripped the Temple of Jupiter Optimus Maximus on the Capitoline Hill of it's bronze roof. However there was still enough decoration for the general Narses to sack a hundred years later.

So the sacks despite their fame and reputation did not cause the large scale abandonment or demolition of the Imperial monuments like the Colosseum or the Circus Maximus. Infact during the 6th Century there is evidence that many monuments were restored. The Ostrogothic King Theoderic made repairs to the Theatre of Pompey, hosted animal hunts in the Colosseum and restored the Temple of Vesta (as proven by a bronze plaque found at the site which states:

With our master Theodoric the Good reigning in Rome [this brick was made])

Theoderic also restored the Imperial Palace on the Palatine Hill which he used as his residence while in Rome. So even after the Western Empire fell we see that there's some degree of activity with these monuments to keep them preserved. In 535 however things in Italy would change dramatically. Emperor Justinian of the Eastern Roman Empire, having just succeeded in reconquering north Africa for the Empire, sets his sights on Italy. His general Belisarius, despite making gains in the first 5 years of the campaign comes to a standstill at the river Po. Rome was initially taken in 537 peacefully, however the city would not have such fortune during the next decade.

The 540's not only saw a devastating plague spread across much of the Mediterranean, but the Ostrogoths saw a resurgence under their new King Totila. During Totila's counter offensive against the Empire Rome would switch hands multiple times. The city would be completely evacuated twice, the Pope fled to Constantinople and many Senators/local officials were taken hostage and/or killed in the chaos which lasted for 15 years.

The destruction of this war can not be overstated, Rome's elite had been eradicated, the population had shrunk to less than 50,000. On top of all of this, the restored Roman control of Italy did not last long and a new, un-Romanized tribe called the Lombards would invade in the 570's which would loosen the Empire's de facto control over Rome.

The city, devastated, with a tiny population and with no concrete central government bureaucracy had it's center almost completely depopulated. It is at this point many of the old Roman monuments start falling into gradual ruin over time, there simply wasn't a big enough population, nor a central government which could sustain the majority of these structures anymore.

But then why may one ask did certain structures like the Forum of Augustus dissapear during the early medieval period while other structures like the Pantheon and Colosseum still survive today? The answer lies in how, or rather if these buildings were still used/occupied and what purpose they served.

The last recorded use of the Circus Maximus is in 549, when the Ostrogothic King Totila, having just recaptured the city from the Imperial forces held horse races there in his honor. However following the war these games became much more difficult to maintain due to their expensive nature and private function.

The Circus Maximus would then over time turn into a white elephant, the city of Rome now no longer had the population nor the government to fund it's primary venue. The Colosseum also stopped hosting animal hunts during this same period because of this, however this is where their stories differ.

Unlike the Circus, the Colosseum actually still found some use during the Middle Ages, although not in the way one might expect. Gladiator fights had been banned in the early 5th Century and animal hunts had ended in the middle of the 6th. The Colosseum was initially following the brief Imperial reconquest closed in order to prevent looters from damaging the structure. However this was seemingly not enforced and looters snuck in anyway, while other people took shelter in the former amphitheatre. The site looked to be heading the way of other Roman monuments, but in the late 6th Century there are records of a small chapel being erected within the amphitheatre.

Parts of the arena were also turned into a cemetery, and many of the arcades started being converted to markets, housing and workshops over time. Later in the 13th Century the entire amphitheatre was even turned into a fortress by the Frangipani.

The Circus Maximus however was not so lucky. The circus was massive and it's position on a flat plain near the Tiber made it subject to flooding, where marshes and malaria would develop. As such the Circus did not find much use during this period.

This is important because all the monuments that survive intact in Rome today, owe their preservation to their later conversion into other uses. The Pantheon was turned into a church, the numerous Triumphal Arches were turned into fortresses etc. The monuments that were not converted were either neglected or used as quarries for new building projects like churches, monasteries and the marble/limestone was even melted down to make quicklime.

This unfortunately was the fate of the Circus Maximus. The Circus found no permanent use, and due to it's large impractical size made ample material for a quarry. When the Circus was torn down is not known, although due to the size of the operation it must have been a gradual process. It isn't until the 1000's A.D that we know the area had been occupied by dwellings.

The Colosseum however thanks to it's semi-continious occupation in some form, and lack of flooding managed to remain relatively intact. Although the amphitheatre also had it's share of misfortune, a rather serious earthquake in 1349 caused half of the edifice to collapse, giving it it's signature hollowed out state we know and love today.

The Colosseum could have infact been demolished during the Renaissance as it's many travertine blocks could and did make for valuable materials for Renaissance projects. Many Popes wanted to either get rid of the amphitheater or find some productive use for it. Pope Sixtus V for example wanted to turn the arena into a silk spinning factory.

What ultimately saved the Colosseum from demolition in the early modern period took place in 1749, when Pope Benedict XIV declared the Colosseum a sacred site of martyrdom for early Christians.

While the Colosseum's history as a early place of martyrdom did not hold much of a historical basis, it's declaration as such saved it from demolition as it's holy status resulted in a ban on quarrying of the site.