There are several reasons, and you might come across the most famous of them through a simple internet research: Christianity. Indeed, even though gladiator games were extremely popular with the Roman plebs (though not as much as the chariot races, which attracted hundred of thousands viewers in Rome and Constantinople), it was the Christian emperor Honorius who, in 404 CE, banned gladiator games within Rome proper (thus affecting the most famous amphitheater of all, the amphitheatrum Flavium, better known as colosseo. So, shifting morals and government action certainly played their part in ending the Roman traditions of games and leading to the abandonment of arenas. (Demandt 2008, p. 356)
But there's another reason; it's lesser-known, but much more interesting, because it helps to understand why a lot of Roman building (including baths, temples and government buildings) were abandoned in the 5th century. The answer lays in municipal finances: Outside the city of Rome, important public buildings like arenas would typically be constructed on behalf of the local authorities: The governor and, more often, the city council. However, the city administration often lacked the money for actual building part, as there was no sufficient tax income. So the city would pressure (or even force) wealthy citizens to perform so called munera (in Greek leiturgiai); such a liturgy could consist in errecting an arena, renovating a temple or providing the wood to heat the baths. Think of these duties as replacements for a not existing extensive municipal taxation.
Of course, as the Roman cities were ruled by the local oligarchs who had the money to perform liturgies, there was a reward for the elites participating in the system: They would earn the gratefulness of their co-citizens for performing their civic duty. Displaying wealth was an inherent part of Roman upper class lifestyle, and necessary to maintain the established social orders. In prosperous times, many would even volunteer (with or without social pressure) to help out their own city financially: this behavior was called evergetism (from euergetes = benefactor). The euergetai would be rewarded with a statue, or, more commonly, an inscriptions (these inscriptions are our main source for the system and help us to understand how local Greek and Roman politics worked). (Christ 2009, p. 386sq.)
The prosperity the empire had experienced in the 1st and 2nd century (and in the 3rd for some regions) had led to the construction of a lot of public buildings like arenas, temples and baths, displaying both the wealth of the city and the liberality of its rulers. These constructions become more seldom after the so-called Crisis of the 3rd Century. On the one hand, the crisis certainly affected the economy in many areas, and the local elites just didn't have money left to invest into their civic duties; on the other hand, maintaining the existing buildings was getting more and more expensive - I've mentioned that providing firewood for the baths was a pretty common liturgy. In a pre-industrial society, seemingly simple tasks could require a lot of energy, energy that just wasn't readily available.
The financial burder of maintaining existing structures was worsened by a couple of phenomenons: First, many cities shrunk during Late Antiquity, which didn't increase their revenue; second, constructing new buildings for your city was still more prestigious than a renovation, so governors or high-ranking city officials would often spend money on new facilities while existing ones needed repair. Even worse, many new buildings were errected using stones from older ones, which was repeatedly banned by imperial law, but to no avail.
Finally, the class supporting this system slowly disappeared as the Roman world fragmented into the Byzantine Empire and the kingdoms of western Europe. The local oligarchies were gradually replaced by new authorities, often bishops, who didn't had to display the same ancient spirit of civic duty. The century-year old system of evergetism and liturgies vanished, and so did the funds needed to finance the maintenance of structures like amphitheaters. (Demandt, 2008, p. 369 sq.; p. 374-377; p. 379sq.; p. 380-382)
In some places like Africa, the local civic upper-class survived longer than in others. These are places were we can observe that the urban aristocracy continued to spend money on public construction; however, in these places, the focus shifted from "mundane" or Pagan buldings like temples, baths or arenas to churches. As new churches were built, money lacked to repair the local aqueduct or to renew the paint in the public baths. Hope that I could help you!
Literature:
Karl Christ, Geschichte der römischen Kaiserzeit. Von Augustus bis zu Konstantin, Munich 2009.
Alexander Demandt, Geschichte der Spätantike. Das Römische Reich von Diocletian bis Justinian. 284-565 n. Chr., Munich 2008.