It's sort of weird right? The idea that it was a temple to all the gods doesn't make much sense, given what I know of Roman religion. What is known of its early use and meaning? What is known or surmised of Agrippa's intent for the original building? What is known of the meaning Hadrian's restoration, that still stands, may have had? Was it even a temple? The dome seems to be intended to be seen, but afaik Roman religion generally didn't have people visit the cella. Was there ever an altar or sacred enclosure associated with the building?
You ask a good question here that has no good answer. The building is a mystery wrapped in an enigma. You mention the Hadrianic restoration, which we must assume was wholesale. 20th-century studies of the bricks from all parts of the building show stamps from the range of 118 to 128 CE, i.e., Hadrian's time, not Agrippa's. The form of Hadrian's building fits well into his time period, as well, though for centuries the peculiar architecture, with the Greek-style porch on the Roman Republican-style rotunda, has been noted. Michelangelo thought it was built by a team of architects with varying levels of education.
We have no clue what form Agrippa's building took, except that it was square and faced south. It was burned at least twice before Hadrian built the current structure on the site. We also don't know to which god it was dedicated, if any. And we don't know what it was called, and the first mention of any name for Agrippa's building ("Pantheum") is from eight decades later. I personally like Ziolkowski's argument that it was originally a temple to Mars in Campo (Papers of the British School at Rome Vol. 62 [1994], pp. 261-277), possibly accompanied by Venus and the deified Julius Caesar.
And a fun fact about this building: most people assume that it survives in such good shape because it became a church early on. This is incorrect. By the 6th century CE, it had been completely abandoned and had been flooded by the Tiber so many times that its floor was thick dirt and silt. There were probably shrubs growing inside. It was not until 609 that Pope Boniface IV consecrated the dilapidated structure as a church.
The inscription on the front is probably the most famous Roman inscription of all. It was put up by Hadrian's architects, obviously, and is no way related to Agrippa. As far as I know, we do not have any record of the original inscription, if it even had one (please someone correct me if I'm wrong about that). The inscription itself (M. AGRIPPA L.F. COS. TERTIUM. FECIT) is very bizarre. Usually, dedications for temples include the god to whom the structure is dedicated, and also use a religious-sounding verb like vovere or dedicare. For example, it might have said: M. AGRIPPA IUNONI / IOVI / VENERI DEDICAVIT. The verb fecit here (3rd person perfect) just means "did it; made it" and is oddly secular.
What Hadrian was doing with the building, why he preserved its association to Agrippa, and why the structure takes its distinctive, odd, and spectacular architectural form is a whole other matter. That's a big enough topic for an entire scholarly book, and I recommend Macdonald, The Pantheon: Design, Meaning, and Progeny (London 1976), who calls it a shine to Rome, to all things Roman, to the Roman Empire, and to the whole world.
You mentioned the concept of public passer-bys being allowed in the cella. This is technically true in the Greco-Roman world, but all the great temples of the Greeks were clearly designed to be viewed and experienced, not shut up behind doors: Artemis at Ephesus, the Parthenon, Apollo at Delphi. The Pantheon was very clearly designed to be looked at, from within, and it must have been open to the public.