Did they organized texts in genres as we do today? Did they distinguished between religious and non-religious texts? Under what circumstances was someone allowed to view, copy, or borrow one of those texts?
I'm afraid the answer here is going to be immensely unsatisfying, but based on the evidence we have, at least in the Roman world, we just can't be sure.
In two letters from Cicero to Tiro, Cicero gives instructions for certain books he is going to send to Tiro: Tiro should put the books away, but, since Tiro had been ill, not write an index until his doctor allows him to (Cic. Ad Fam. 16.18; 16.20). Cicero’s letter comes tantalizingly close to answering all sorts of questions about Roman book collections, but still leaves us with quite a few. There is going to be, when Tiro is feeling better, an index, a list, of the books in this collection, which means they are being tracked in some way. But Cicero doesn’t mention — Tiro, it would seem, already knows — how this index is to be organized. Cicero’s letter is characteristic of much of our evidence on the organization of libraries in the Roman world: it gives us a hint of a glimpse into something, but it ultimately leaves us with more questions than answers.
For the Roman world, book lists are one of few pieces of surviving evidence we have for organizational activity inside libraries. Unfortunately, although those book lists — recovered exclusively from Roman Egypt — give us a view into the collections as a whole — the lists often acquire a distinct personality based on their distinct focus on philosophy or medicine or epic — they don’t tell us anything about organization within the collections. They include no information about how to find a work in a collection or any of the other information you might find in a modern library catalog. They aren’t consistent in their organization. Some are alphabetical by author or title, but not all of them are. Some lists keep poetry and prose separate, others don’t. Some seem to be divided at least somewhat according to genre or subject matter, but many are not. Some aren’t ordered at all. We don’t know if these lists are final lists or, like the index Cicero asked Tiro for, partial, or simply inventories made to a collection that would later grow. We don’t have their beginnings or endings, so the lists we have are not complete, and we don’t know if they were titled. Because they were incomplete, we cannot be certain in one case about a list (P.Oxy. 33.2659) that Houston speculates may have belonged to larger, non-private library (in addition to the public and municipal libraries, Martínez and Senseney have argued for the existence of special libraries run by guilds or professional groups). Given that the list in question includes a detailed coverage of Old Comedy and nothing else, knowing where it came from would help us answer a number of questions and probably ask a few new ones. All of that is a lot we don’t know about book lists, especially written out like that.
Our literary evidence is similarly lacking, although book borrowing is mentioned some. From epistolary correspondence between Marcus Aurelius and Fronto, it would seem that the emperor’s heir was able to borrow books from the public libraries, but that the same was not true for anyone else. This was not necessarily the case for other libraries elsewhere, but we can’t know one way or the other.
Literary evidence shows one way one might access books, but still doesn’t answer questions about the organization of the library. In references to visiting libraries found in Aulus Gellius’ Noctes Atticae, none of the characters actually interact with the shelves themselves. We know that rolls were labeled with tags that gave its contents, but none of Gellius’ characters use them at all. Slaves working at the library retrieve books for them, delivering them to the characters while they are seated. Presumably the slaves working in the library knew the organizational system, since they had to be able to quickly retrieve the correct author and work, but there is no catalog system to access because, as Houston puts it, “the slave is the catalog”. The slave might be able to guide you to new materials, but Aulus Gellius doesn’t show his characters browsing the shelves themselves or exploring a shelf on tragedy, for instance. Their access to the library’s organizational system is indirect and mediated, and in the story as told in the Noctes Atticae the library, its shelving, and the slave actually giving them access to the materials all disappear from the narrative, almost as if books present themselves of their own accord. They don't, but Gellius doesn't consider the system doing the collection and organization of the knowledge he's accessing to be worth mentioning.
Houston, George W. “How did you get hold of a book in a Roman library?: three second-century scenarios.” Classical Bulletin 80.1 (2004): 5-13.
_______________. Inside Roman Libraries: Book Collections and their Management in Antiquity. Chapel Hill: University of North Carolina Press, 2014.
Martínez, Victor M. and Megan Finn Senseney. “The Professional and his Books: Special Libraries in the Ancient World.” In Ancient Libraries. Edited by Jason König, Katerina Oikonomopoulou, and Greg Woolf. Cambridge: Cambridge University Press, 2013.