More can always be said, but meanwhile check this answer from /u/Alkibiades415
In addition to all the correct comments made above, I think another point worth of being stressed is that the conservation, restoration and valorization of the historical and cultural heritage, particularly for buildings and monuments, at least as we conceive it today is quite a recent concept. During ancient times, the Colosseum continued to be restored until when it maintained its original and primary function. When its original function ceased, to maintain it in its original state was not only impossible due to the overall crisis of the early Middle Ages but was also substantially useless, a waste of 'public' funds. It was much more reasonable to use it in compatible ways with its condition.
To say it in other words, today when we look at the Colosseum we see the main touristic attraction of Rome and one of the 7 New wonders of the world. For many centuries, Roman citizens saw an amphitheater in decay with no prominent function.
Connected to this, it must always be reminded that the popes governing Rome were always prone to commission extraordinary monuments, churches and buildings, but their mission was to celebrate and foster the magnificence of Christianity, and in particular of the Holy Catholic, Apostolic, Roman Church (in addition to leaving a notable trace of their existence). Colosseum could be very hardly involved in any such reasoning. Accordingly, the first solemn and prestigious use of the Colosseum from ancient time happened when a Pope, Benedictus XIV, recognized it as a holy site for Christianity, due to the alleged martyrs perished there.
Rome and other big cities of the Roman Empire were depopulated during the Dark and Middle Ages. Without an imperial army and navy to protect trade routes, banditry made travel and transport of food and other goods difficult, dangerous and costly.
Lawlessness also affected the great agricultural estates owned by the wealthy Roman aristocracy. Large holdings became impossible to manage and protect. Most were divided into small plots of land and sold off to peasants. Subsistence farming became the mode.
Urban public works like the Colosseum could no longer be supported by taxes on agriculture and trade. During the Empire, important, wealthy men had been expected to donate monuments, temples and other public works to the city. As their wealth dwindled, so did such philanthropy.
By the 6th century, there was significant deurbanization. The resources needed to maintain a vast system of roads and aqueducts, as well as stadii, theaters and other public structures, were no longer available. And there were no great crowds to attend the games and contests. Over time, grand structures like the Colosseum fell into disrepair - and most remain that way today.
There are different issues to consider in order to put the answer to your question in perspective.
First and foremost, it is important to understand that the concept of Historic Monument is quite recent: the idea that some constructions or objects are important in a strictly material way as some kind of testament of the must that must be preserved untouched has not operated in history untill the modern age. Of course People in the past gave great importance to monuments and sometimes took pains for the sole purpose of experiencing them, but they did so in terms of the whole thing: the material reality itself of the structures was not the main point. Romans, for instance, had no problem rebuilding lost buildings again in diferent styles and shapes, even when it had been posible to build them back as they were, the main example being the Pantheon in Rome, which has a dedication copied from the old, destroyed Pantheon, even if the new one was a totally different buildings. Staying in Rome, the most central temple in Western Christianity, (old) Saint Peter's Basilica, constructed by Constantine himself, was demolished so the new one could be built. The decision was surely controversial even if the building was very old and in a poor state, but regardless the new basilica was started and very little material from the old one was preserved. A lot of the more useful marble and columns and doors were reused, but even pope's tombs were destroyed for material or just discarded. Only during the last stages of demolition did the pope try some kind of documentation process of the artifacts destroyed and some tombs were moved elsewhere. So, a Pope removing ancient pope's tombs from their sacred resting place gives you an idea about how absent "historicity" and the importance of preserving material objects based on their antiquity was in past people's minds.
It is important to keep this in mind so we can understand the processes ancient buildings have exoerienced untill the present. For most of their existence, the Flavian Amphitheater and other Roman monuments were not "monuments", they were material realities that were there to be used in whatever context people using them operated. The same way a medial Roman person would not see the tiber as an important part of a big natural ecosistem encompassing the planet and with an inherent value forc that that merited protection and preservation, but as a source of water, transport and death, they would not see the Colloseum or the forum as monuments but as places to do things in and get things from. Some buildings were destroyed by accidents, climate, natural disasters or abandonment, but most were reused, whether as a source of material or as a source of resignified space. In fact, one could argue that a lot the most splendid monuments of Rome survived more or less the middle ages, as some of them got destroyed by post reinassance elites for marble.
So now, about Rome being continuously inhabited. It is true, but with a lot of fluctuation. First we have to consider the literally exceptional nature of Roman population. It was made only possible by a huge system of water, grain and people imports. The population could not even maintain itself, it needed a constant influx of people. What I am trying to get as is that having half a million people living in an urban environment in premodern times is extremely difficult, and the infrastructure required extremely complex to manage and maintain. Rome lived on imported grain from Africa and Egypt. It was locally managed by an extremely wealthy class under a system that incentived private spending on public works, making Rome herself even more prone to monumentalization that it was infraestructurally necessary.
So, contrary to the old and disproved "dark ages" narrative, Rome as a city managed relatively well two sacks and the collapse of the western emperorship. After the reforms of diocletian and Constantine, that not only moved political power east but destroyed the political relevance of the old (mainly western based) senatorial aristocracy and made Nova Roma Constantinopolitana the new center of political power, Rome lost its preeminence in imperial politics but maintained the same old structures of power and largesse. Emperors kept building stuff and maintaining stuff, at some point churches even. Some of the most impressive remains in Rome in terms of monumentality, like the bassilica of magentius or the baths of diocletian were built during this era of political non relevance. Even when there were emperors in the west they would not be in Rome but in Trier, Milano or Ravenna. So when the goths took over and kept the socioeconomic system in place Rome didn't even notice: the new Gothic kingdom even maintained the last imperial residence of Ravenna as Capital. What was left of the old Roman senatorial class still hold local power and managed the city, they were still extremely wealthy and had the same ideological use for monuments such as the Amphitheater or the Circus, so games and spectacles were given to celebrate consuls and traditional days well after the "fall of the empire".
Edit: phone typos