I believe you are confusing the Pantheon with the Parthenon.

The Pantheon, in Rome, is indeed partially made of concrete. The aggregate in the concrete of the famous dome starts with very dense stones at the base ring and it gradually lightens in density as you move up to the oculus. But back to your question - it's in excellent condition for a 2100+ year old building.

On the other hand, the Parthenon, on the Acropolis in Athens, was partially destroyed in 1687 after standing mostly intact for a little over 2000 years. For what it's worth, the building is made of marble - not concrete - not that this would have a big impact on what happened to the structure: during the Morean War, the Ottoman Turks were using the Parthenon as a powder magazine and the besieging Venetians landed an (un)lucky artillery shot that blew the roof off the building and destroyed the cella (which is the actual enclosed room of the temple) along with several columns, etc.

Wikipedia has decent information for the architectural topics here, but if you want to go into more detail, Trachtenburg's book Architecture: From Pre-History to Postmodernism, is a great starting point. It's been used at so many architecture schools over the years, you can get a used copy dirt cheap online. The only downside with it - I wish more photos were in color.