There's an element of chance in how any ancient document is preserved. Cicero's Republic, for instance, although well known in Antiquity (we knew about it because people talked about it) was lost in the Middle Ages until 18...19? when it was discovered as a palimpsest - some monk had erased it and written another text on top of it, but with careful reading the original text can still be made out. That's the only manuscript of that work that survives, aside from the Dream of Scipio bit at the end, which has a separate manuscript tradition.

It's kind of an instructive example. It tells us that someone valued the material the Republic was written on more than the text of the Republic, and preferred to use that material to preserve another text. In this case it was something of Augustine's, so a religious text. The Republic was lost for hundreds of years not through any malice, but simply because people were not interested in reading, and so copying it.

This has happened to a number of books we know existed and we would very much like to have. I myself would love to have Sulla's commentaries. They were known and quoted by Plutarch about 200 years after they were written, but since then we lost them. It has nothing to do with being popular or well-known in the ancient world and seeming important to us now. At some point a failure to produce new copies of these works means they go away.

Claudius' books were subject to the same forces as The Republic or Sulla's commentaries; they were not thought at some point worth reading or reproducing, and so they ceased to exist.

Every so often a new text pops up - there are discoveries of papyri in Egypt from time to time (mostly fragments but sometimes extended texts), I'm sure there are undiscovered palimpsests, the Villa of the Papyri contains a library of burned books which I understand modern technology is able to read. No way to predict what will turn up or when though.