Well it's really a combination of several factors, not all of it tied to construction material.

First, is in fact the construction material. As I'm sure you're probably aware, the Romans were some of the pioneers of concrete construction. Many of the very large structures in Rome, like the various baths or the pantheon were constructed out of Roman concrete and so were very durable. However, these tend to be public or aristocratically financed buildings. You're never going to see the survival of anything made out of more biodegradable material unless it's maintained regularly and continuously (like for example Chinese and Japanese Buddhist monasteries with continuous inhabitation).

However, there's a second, less obvious reason for the survival of Roman buildings. It's because there was a massive economic decline in the aftermath of the collapse of the mediterranean Roman empire which led to the abandonment of those buildings, without the economy for re-occupation until nearly a millenium later, when there was antiquarian interest in preserving those ruins.

There were some very well constructed monumental Roman buildings in the eastern mediterranean, like in Alexandria for instance. However Alexandria and many other cities in the east had nearly continuous occupation, so once large monumental buildings were abandoned and ceased having purpose, those building sites were reclaimed. Either the structures were used as quarries for new buildings, or they just removed those old structures and built new ones atop it. This reclamation of course has no bearing on the solidity of the building structure.

What this means is when you see a long-standing Roman building, it either has a history of continuous occupation (Pantheon) or it existed in an abandoned older part of a shrunken medieval city (Baths of Caracalla), or it was excavated later and preserved because of antiquarian interest (The Roman Arena in Paris comes to mind). If the city had continuous inhabitation across the entirety of the period, especially across cultures that had no antiquarian interest in their Roman past (like Islamic mediterranean cities), those structures wouldn't likely survive. Which is why in Alexandria today, the parts that were Roman that do survive are the ones that were excavated in the last few hundred years, not those that have survived because they were somehow better built.

In the case of buildings that didn't survive intact, but were built of lasting materials, in some cases the cut stone was "cannibalised" by later inhabitants of the same area to build new things. For example, if you visit the area around Hadrian's Wall in the UK, often the old village churches in local villages may have incorporated stones from the Wall or Wall forts. Sometimes they have inscriptions on them and you can have the fun of reading a standard Latin dedication to some god or other, sideways, half-way up a wall.

Stone is pretty permanent. /u/bitparity outlines why some stone structures don't survive, but surviving is pretty much the default state.

Stone buildings use stones joined with mortar. Stone does not rot or burn like wood, it does not rust like metal, it does not react with the environment like concrete. Stone just sits there.

Mortar, however, is concrete-like and does react with the environment over the course of a century or two, but it ultimately isn't that important: gravity and friction do most of the work for keeping a stone building in one piece.