Essentially, no.
Architecture history is full of examples where a building was torn down, either for the building material, or the space it occupied, in order to make a new, different building. Roman architecture that you mention in your answer is a good example of this, as a number of buildings from that period have suffered this fate, serving as impromptu cheap and efficient quarries. Due to a continuation of urbane composition even after the fall of the Roman Empire, we can still read a number of roman urbanistical elements – most usually the cardo and decumanus which still exist as main communication areas in a number of european cities. The buildings that defined the cardo and decumanus haven't stood the test of time, however.
You're mentioning the Colloseum in your question, which has an interesting story to it; in 16th century, Pope Sixtus V had plans of turning the derelict roman building into a wool manufactury. Once the budget of such an investment was deemed unfeasible, this was abandoned, but Pope Benedict XIV in 18th century forbade the building to be leveled, due to it being (wrongly) seen as a place of martyrdom of christians who were 'fed to the lions', this false historical anecdote saving the building from being demolished. In other interesting mentions of roman amphiteathres being converted within an urban context, one of the most interesting is that of the amphitheatre of Lucca, at the so called Piazza del Anfiteatro, where a building block was formed in the amphitheatre's abandoned structure.
Piazza del Anfiteatro, Lucca, Italy.
Another important event which brings us closer to understanding how our ancestors percieved building heritage ( which is essentially, they didn't ) was quite common in the baroque period – a large number of middle european gothic churches were quickly 're-baroqued' in this period; as gothic style was seen as a barbaric, medieval, primitive ornamental style, a lot of its building elements were intentionally torn down or walled in, in order to make place for the new, baroque elements. This was the first occasion that building heritage was destroyed for the sake of 'style', instead of raw pragmatism of easily accesible material like in previous periods.
Awareness of building heritage as we know it today stems from the 19th century, development of academia, and architecture history as an academic discipline. This has to be seen through the dichotomy of theories of Eugene Viollet-Le-Duc and John Ruskin. Whereas Viollet-Le-Duc argued that historical buildings deserve to be 'creatively' finished, to look even better than they ever were in their history, even if the elements that are used are not exactly original ( and did well on his theory by creatively destroying Notre Dame de Paris and Carcasonne ), Ruskin in his pivotal work, Seven Lamps of Architecture argues that building conservation must first and foremost be honest; and to show all the layers and elements of architectural progress in all of their parts and history, instead of presenting a final 'candied up' finished product, and not imagine things up without solid historical proof.
Later, mostly through the work of Austrohungarians Alois Reigl and his student, Max Dvorak, conservational criteria and principles of John Ruskin have been shaped into architectural heritage conservation policies which we're still abiding to to this day, but Ruskin is widely seen as a 'father' of the contemporary building heritage conservation method.
Before Ruskin, there was just simple destruction at best, and reinterpretation and making up of history ( like Viollet-Le-Duc ) at worst.
The second reason is, that imperfection is in some sort essential to all that we know of life. It is the sign of life in a mortal body, that is to say, of a state of progress and change. Nothing that lives is, or can be, ridgidly perfect; part of it is decaying, part nascent. The foxglove blossom,--a third part bud, a third part past, a third part in full bloom,--is a type of the life of this world. And in all things that live there are certain irregularities and deficiencies which are not only signs of life, but sources of beauty. All admit irregularity as they imply change; and to banish imperfection is to destroy expression, to check exertion, to paralyse vitality.
Sources :
John Ruskin, Seven Lamps of Architecture, original publication 1849.
Jukka Jokhileto, History of Architectural Conservation, 1999.
