Buildings and monuments are built to serve certain purposes, individuals, and ideologies that are very specific to their time context. More often than not those purposes, individuals, and ideologies are forgotten with time, or sometimes they might even gain new negative connotations, and thus those buildings become obsolete. The reason why we today (in Western Europe especially) spend billions on excavating and preserving historical monuments is because our age is marked by an ethos that the past in itself has inherent value and is worth protecting and transmitting to future generations - but this is a fairly "modern" ideology that has its roots somewhere in the values of European Renaissance and Enlightenment (other people here probably now more about this than me), but the zeal for excavation and restoration of historical remains did not kick off properly until the 19th century, when Western Europe became steady and wealthy enough that people could start devoting time, energy and resources to excavation instead of just survival. These sort of conditions, the right values and right resources, haven't coincided that often in European history.
The history of the Roman forum might be quite illuminating. During the High Empire, the Roman forum was, as it had been for centuries, the administrative, religious, and ideological heart of the whole Empire. In the Forum there were important political buildings, such as the Curia, the meeting place of the Senate, government buildings, and the Rostra, platform for public speeches; some of the most important temples, such as the Temple of Vesta and temples of imperial cult; and honorific monuments celebrating the history and achievements of Rome, such as triumphal arcs and statues of foremost citizens. Everything suitably grandiose and lavish, of course. The forum was in daily use of business, politics, and various celebrations.
The status of Roman forum started to decline when during late 3rd and 4th century Rome was stripped from its status as the administrative capital of the Empire and the Empire became officially Christian. Late antique emperors ruled from Milan, Ravenna or various Gallic cities. Interestingly, the Late Antique Roman society still appreciated its triumphant past, and although all substantial state investment on the Roman forum stopped, the local elites still spent money on costly restoration projects of the forum to prop up their own status (Gregor Kalas´ The Restoration of the Roman Forum in Late Antiquity: Transforming Public Space is a great book on this period).
However, with time, the "pagan" past was no longer straightforwardly just glorious and great; statues and monuments to great Romans could be seen as heretic idolatry; some of the monuments were converted to Christian churches. With the collapse of the Western empire (last Western emperor murdered in AD 480), cohesive imperial ideology was more or less lost and the glorious past of the Empire mattered less and less to the locals. The city of Rome also entered a long period of decline: the population was cut to perhaps as low as only 25% from what it had been at its height, and the Germanic invasions massively damaged many of the monuments in Rome. So, at this point, there really was no any clear ideological reasons to restore historic monuments - the elite of Medieval Europe mainly spent their money on patronage of Christian churches, communities, and monasteries - and nowhere near as much expendable resources. The literature of Greco-Roman antiquity and the knowledge of Roman past was also not as important as it had once been.
Once the buildings became more and more shambled and ruined, the less people of course remembered and cared. 8th century travel records already have reports of the forum as being a sad ghost ruin of what it once was. Lot of the monuments of the Roman forum were pillaged and looted for easily available building materials for churches, towers, castles and other more immediately important, new constructions. The only reason why the Pantheon in Rome has been preserved in its amazing more or less original glory is because it was fairly early on (7th century) converted to a Christian church, and it has been in continuous use as such ever since. Debris from crumbling buildings in the Forum and around it raised the ground level and further buried the remaining ruins, and at some point it was really only used as a dumbing ground. By 17th century the Roman forum was known locally not as the Forum, but as "Campo Vaccino", "the Cow Field", as the debris and dumps had turned into a little open field in the middle of the city where locals let their livestock graze, with only few sad pillars and triumphal arcs marking the landscape.
When during Renaissance and Enlightenment the antiquity was "rediscovered", many artists were inspired by this landscape of decaying ancient glory and you can find lot of paintings and drawings from the Roman forum; see as examples Claude Lorrain, 1636, Giovanni Battista Piranesi, mid-1700s,Charles Lock Eastlake, c. 1820s. Just shows how little there was left! It wasn´t until late 19th century, when archaeology as a discipline had been developed and become into "high fashion", that official excavations and restorations of the forum started, and during the 20th century in various stages the forum has been restored to as you can see it today.
e. typos fixed
Dumb question, something I was wondering about recently. Why do ancient developments “sink” into the ground? Many times ancient buildings, roads or entire cities are discovered underground. Will our currently existing infrastructure be underground too one day? Sorry, haven’t had coffee yet.
I can offer a specific case study from Central America, that I think will illustrate how this sort of change can happen, leaving large ruins in the middle of modern cities. Note that all names given here are modern names for places, and in most cases don't reflect their Prehispanic names, which are unknown.
Back in the Early-Middle Preclassic period (around 1200-900 B.C.), a small village was settled near a pond in what is now El Salvador known today as Laguna Cuzcachapa. This village grew, and eventually, a group of large civic/ceremonial buildings was built a short distance to the north, centered around a big earthen pyramid known as El Trapiche. At the time, this community would have been one of a string of large towns/small cities strung out along the Pacific Coast of Chiapas (Mexico), Guatemala, and El Salvador. The Trapiche pyramid itself (built and used between 900 and 650 B.C.) was one of the largest buildings on the Pacific Coast at the time, and was surrounded by other structures and by stone monuments that link the settlement to the Olmec cultural tradition, which was centered on the southern part of the Gulf Coast of Mexico. With the change in focus of the settlement to the north, the small village around Laguna Cuzcachapa would have fallen into disuse and become a series of small rubble mounds on the edge of the town. The focus of the town remained around El Trapiche until the late Preclassic (maybe until around 200 B.C.).
At the very end of the Preclassic period, a large volcano (Ilopango) located nearby erupted, disrupting agriculture and trade on the Pacific Coast. It's likely that the settlement around Trapiche declined in population at this time, and the big pyramid was probably abandoned.
About 200 years later (A.D. 1, roughly), the settlement began to grow again. Now, though, it was centered around a new ceremonial center called Casa Blanca, a short distance south of El Trapiche. The Trapiche pyramid and other structures nearby would then have stood at the north end of the settlement, and would have looked like large earthen mounds, probably covered with grass and brush. We have no idea, of course what the later people would have thought of these structures (already pretty ancient), but they probably would have recognized that these were pyramidal platforms not unlike the ones in the new Casa Blanca complex. This new settlement was linked in to Early Classic events around them, and may have spoken a Mayan language, as this part of El Salvador seems to have been in the Maya cultural area throughout the Classic Period.
During the Early to Middle Classic (A.D. 200-650), the focus of the settlement began to shift again. A new focus to the settlement was established at Tazumal, to the southwest. A new large pyramid (Structure B-1, or just El Tazumal) was built, not as tall as El Trapiche, but larger in volume. Other structures in the Tazumal area include a smaller pyramid built in the talud-tablero style (possibly associated somehow with the Teotihuacano culture of central Mexico). By the Late Classic (A.D. 650-900), Tazumal was the center of the community and Casa Blanca was abandoned, becoming a group of large ruins, presumably toward the eastern edge of the settlement.
Things in the region changed a great deal in the Early Postclassic (A.D. 900-1200). New people, speakers of Nahuat, a Nahua language closely related to Nahuatl, the language of the Mexica (Aztecs) began to migrate into the area, and may have established new towns in previously unoccupied areas (though this interpretation is controversial). However, at Tazumal things went on more or less as before, despite the abandonment of nearby Maya-like towns such as San Andres, and other settlements in the region, such as Cara Sucia (which is apparently affiliated with the little-understood Cotzumalhuapa culture). We know the settlement continued, because the large pyramid at Tazumal was rebuilt several times through the Late Classic and Early postclassic. In the later part of the Postclassic (A.D. 1200-1500), something changed again, and Tazumal was abandoned, and a new group of much smaller ceremonial structures were built in an area known as Peñate. These smaller structures were the center of an apparently smaller settlement affiliated (based on Spanish accounts) with the Pokomam Maya culture of highland Guatemala. At the time of the Spanish Conquest of El Salvador in the 1520-1530s, this Pokomam village would have been living surrounded by ruins of large structures dating back almost 3000 years.
After the Spanish conquest, a Spanish presence (and a church—still extant) was established in the Pokomam village, which is now known as Chalchuapa (Santa Ana Department, El Salvador). The village lost its Native American character and became a ladino community (that is, a community of people of mixed Spanish and Indian descent, who participated in Spanish Colonial culture). Slowly, the village grew into a town, and then into the modern day small city of Chalchuapa.
As the community grew, it started butting up against the ruins all around. By the 1980s, many structures had been leveled or partially destroyed, in many cases for building material. The small structures in the Peñate group appear to have completely destroyed in the past 30 years or so. However, the three big complexes at Tazumal, Casa Blanca, and El Trapiche were preserved. The two former groups were subjected to heavy looting in the 20th Century (particularly at Casa Blanca), indicating that people recognized these as the remains of ancient cultures. A gravity-feed water tank has been standing on the top of the Trapiche pyramid since at least the late 70s, and there is a small commercial swimming pool facility (balneario) at the base of the structure. Unlike Casa Blanca and Tazumal, it appears that many people thought that El Trapiche was just a natural hill, not recognizing it as an ancient ruin.
Today, Tazumal and Casa Blanca are archaeological parks, partially restored and open to the public. The balneario at El Trapiche is still there and still open, to the best of my knowledge, and the site is monitored by Salvadoran government archaeologists. Recent growth of the city appears (based on Google Earth images) to have largely overrun the area of the very early settlement around Laguna Cuzcachapa. Many smaller structures around both Tazumal and Casa Blanca have been wholly or partially destroyed, and the city cemetery lies partially within the Classic-period ballcourt.
So that’s a brief case study of how one settlement continued to grow around its own ruins, which became progressively more ancient.
The main source (other than some observations of my own in modern Chalchuapa), is:
Sharer, Robert J. (editor) 1978 The Prehistory of Chalchuapa, El Salvador. University of Pennsylvania Press, Philadelphia.
u/Mythoplokos had brought up a wide array of very valid points; with the decline of Rome's political and economic importance, the utility of maintaining the forum was not immediately evident to Rome's declining population.
I'd also like to point out that during Rome's subsequent re-population, the center of population shifted about a half-mile to the north, closer to the post-roman ecclesiastical center around St. Peter's basilica and (more extensively) the area directly across the Tiber. But while the Forum was now on the fringe of Rome's most densely populated area, it fate of the Forum isn't entirely representative of the totality of ancient buildings in the city, with many others repurposed like the Basilica of St. Mary of the Angels and the Martyrs (Santa Maria degli Angeli e dei Martiri) built inside a part of the Baths of Diocletian, while the Colosseum itself was incorporated into a fortress complex (as the Mausoleum of Hadrian was converted into Castel St. Angelo) and while we don't have clear reports of its gradual deterioration, it's hypothesized that conflicts in the early middle ages where the structure was a military target did more to damage it than centuries of neglect.
The present urban texture of the Roman Forum is a consequence of very modern phenomena: on the one hand a modern appreciation of history and archeology has made it desirable to keep the ruins visible, while on the other hand the modern sprawl of the city of Rome means that the forum is once again surrounded by an urban environment. While estimates of the city's population at the height of the Empire run anywhere between two hundred thousand to one million, the city would only begin to approach the lower tail of these estimates halfway through the nineteenth century (and surpass the upper tail sometime in the 1930s). While the elegant apartment buildings in and near the Forum largely blend in with the city's earthy tones and mixed baroque and neoclassical architecture, many if not most of them were built in the decades immediately preceding and following the turn of the 19th century; it is more accurate to imagine the ruins of the forum to be set against a much less dense backdrop for much of the period between the end of the Empire and the modern day.
While I can't speak too much about Athens (although u/Tiako's answer, linked elsewhere in the comments, goes into great detail about how the immediate area of the Athenian acropolis was much disencumbered of more recent constructions after Greece's independence) Rome's extensive areas of open-air ruins are an anomaly in Italian urbanization. Very few other Italian cities experienced depopulation and outright relocation the way the imperial capital did (Ravenna comes to mind as one of the few similar examples, although the ruins in Ravenna are exponentially less extensive than Rome's and its monuments are largely still intact, albeit awkward and out of place a millennia after the city's brief tenure as Italy's political epicenter expired). In other cities, like Brescia, open spaces with ancient ruins in the city center are the consequence of a meticulous and sometimes ongoing process of excavation, demolition, and restoration which can create awkward questions if structural elements from the Roman era have been incorporated into medieval buildings, forcing the local archeological team to decide what they value more (an intact roman colonnade or perhaps the palace of an important medieval dynasty?). The point is that it is much more common to repurpose and build on top of structures, as the Milanese did with the Basilica of St. Lawrence (Basilica di San Lorenzo), than would be to leave open space with ancient ruins as is in Rome.
