While it seems really abrupt to us today, Rome's decline from the greatest city in the Mediterranean to a city of ruins took centuries to occur.

Rome was a city with great construction projects up until the middle of the 4th century when works like the Arch of Constantine and many of the city's great ancient churches (The Lateran, Old St. Peter's, etc) were constructed. With the Western Imperial capital at Milan and after Constantinople was built, the focus of new imperial constructed shifted north and east and additions to the city of Rome became rarer. Ammianus Marcellinus notes the pomp and festivity surrounding Constantius II's gift of an Egyptian obelisk to the Circus Maximus in 355.

As conditions in the West became more chaotic in the 5th century, Rome was twice sacked, by the Visigoths (410) and the Vandals (455). The city was looted of its most valuable gold and silver, and the damage caused by the subsequent fires and vandalism (pun not intended) were only mildly repaired. On top of that, the Roman generals often made use of Rome to supply their troops. Stilicho, the Roman generalissimo between 395 and 408, had some buildings stripped of their precious metals to pay his troops to defend Italy. With the Vandal taking of North Africa in the 430s and 440s, the free grain dole was discontinued and Rome could not maintain the large urban population that it had for the previous 5-6 centuries. Rome's population shrunk dramatically.

The city was not in complete ruin though and a large part of the city still functioned well into the 530s when Justinian attempted to take Rome from the Ostrogoths. The Gothic Wars (534-52) saw Rome switch hands 5 times between Byzantine and Gothic armies. The historian Procopius noted the aqueducts were cut and never restored, the countryside around Rome devastated, and even the Senate as a body ceased to meet. At this point only the Church seemed to operate undisturbed. By the 7th century it was considered a big deal when the Roman elites commemorated a pillar, yes just a pillar, to the Emperor Phocas. And the pillar was from a older, now ruined building. In the 660s the Emperor Constans II visited Rome and after addressing the 'senate' he had his men scout the city for whatever valuable materials remained to supply his army.

At this point, Rome was used as a source of marble for the surrounding region and for repairing the still-in-use buildings in the city. Rome had shrunk to a small pocket of what had once been the great city, and few objected to the cannibalizing of its many, many ruined buildings. Charlemagne famously built his palace at Aachen out of Roman marble.

Funny enough, the Romans tended to not harm the buildings that were converted to Church's, hence why the Pantheon and Hadrian's Mausoleum were relatively untouched.

As for medieval Rome, I'll have to defer to someone with more expertise.

Sources: Ammianus Marcellinus, Res Gestae

Procopius, The Gothic Wars

Chris Wickham, Early Medieval Europe, 300-1000

Warren Treadgold, History of the Byzantine State and Society

EDIT: As u/shlin28 noted, when Constans II visited in the 660s he met with clerics in the city, and not the senate. But he also met with the new tiny body of Roman elites who referred to themselves as the senate but as rhetoric more than actual reality. Hence my quotation marks around the word 'senate'.

This thread doesn't answer your question directly, but it is one of my favourite replies on the sub: u/shlin28 on Suppose you visited Rome in A.D. 361, and then in A.D. 661, what difference would you see?

The reply does a great job of bringing the city to life, giving you a feel for what it might have looked like and how it changed, beyond what graphs and figures tell us.

Incidentally, although this is beyond my own area of expertise, I note that u/shlin28's post disagrees with the graph you link to, giving Rome's population in the mid 4th century as around 800,000 people. Probably somewhat more than half of what it had been in its glory days, but far less dramatic a fall than the graph depicts.

May I ask where you got that graph from? The extremely sharp decline in the 3rd century seems rather dubious, to say the least. Rome's population fell to (considerably) less than 100,000 eventually, but not as early as the year 300 A.D.