As with lots of buildings in Late Antique Rome it's hard to know exactly what was going on at specific points. The literary evidence is patchy and the archaeology isn't always much help either. Nevertheless, there are some things we can say for certain. Unlike the Pantheon, the temple of Jupiter Capitolinus (the one I assume you're talking about) was never converted to a Christian church - perhaps it was too symbolic of pagan Rome to be safe for that kind of thing.

By the fifth century there would have been no worship at the temple - laws passed under Theodosius at the end of the fourth century were supposed to have put an end to such things, and certainly there were fewer and fewer active pagans in the city even before then (not to say there were none, and in fact a group of aristocrats remained quite visibly pagan throughout the fourth century).

Zosimus (5.38) says that Stilicho stripped the gold from the doors of the temple in 409-10, when Alaric invaded Italy. The year after your visit, in 455, the Vandal king Gaiseric would also strip half of the gold tiles from the roof (Procopius 1.5). Overall I think you would see a closed building, still with some of its decoration in tact, but neglected and perhaps beginning to fall on hard times. To make yourself feel better, I would recommend heading over to the esquiline hill to marvel at the basilica of Santa Maria Maggiore, only a decade or so old at this point and very beautiful!