Books were taken down by hand, whether they were in the form of a 'book' or a scroll, and thus immensely expensive. The primary reason libraries existed was because copying was such a difficult process, and thus scholars needed a place where they could have access to these works.

The other thing to consider is that there was no formal system of education, and thus, at best, we can expect literacy rates to have been well under 50%, and probably a lot closer to 10-20%. Exact numbers are pretty much impossible, so we mostly have to guess. The rich and even moderately wealthy would all have been trained in letters, as this was a pre-requisite for any sort of government work or senior position of any kind in the Legions.

Speaking of which, the vast majority of 'books' in the Roman Empire were not private works on philosophy or history, but government documents. The Roman Empire was bad at a lot of stuff, but one thing they were exceptional at was record keeping, and the output of the military and civilian apparatuses in keeping records of service, land, and law were extensive.