So in Europe the most common system was based on shelfmarks that gave the location within the library. Using the early modern Cottonian library as an example, we have a system of shelves distinguished by having a different Roman bust on each shelf, followed by shelf number, followed by location on the shelf. So for example the Lindesfarne Gospels( Cotton Nero D.iv) would be the fourth book, second shelf, under the bust of Nero. The use of a system like this is that while you don't automatically know how the library is organized you can find any book geographically given only the call number.
Well, after a long time of lurking (and realizing that I've lost the password to my old account) it is time to come out of the shadows with a new account and talk about medieval libraries.
A first point to note - asking about libraries in the "Middle Ages" involves over a thousand years of book history, roughly the amount of time that has passed since the central Middle Ages until the present and around twice as much time as has passed since the invention of printing. So there is no single way in which libraries were organized in medieval Europe.
But let's look at a few examples. Folio 5r from the Codex Amiatinus (google Codex Amiatinus) has a nice image that is often taken to represent part of a medieval monastic library: some books in some sort of cabinet, kept in part of the cloister. This particular image has only about ten books, which might be all that a small community would own. In this case, no real organizational system would be necessary.
By the eleventh or twelfth centuries, the most prestigious monastic communities might have manuscript collections numbering in the hundreds (c.f. N.R. Ker, Medieval Libraries of Great Britain: A List of Surviving Manuscripts). But shelfmarks, even at this stage, were relatively rare. So far as we can tell, books would have been organized in a series of cabinets roughly thematically, with scriptural texts first, possibly with liturgical materials alongside them (although these might have been kept separately in the sacristy). After that, patristic texts, followed by histories, theology, secular literature, and other genres in some order. Sometimes basic texts on the liberal arts might have been kept in a separate location as well. Most of the evidence for this organizational schema derives from medieval booklists (e.g. Albert Derolez, Corpus catalogorum belgii: Medieval Booklists of the Southern Low Countries or any of the volumes published in the Corpus of Medieval English Library Catalogues). But even then, most complete catalogues of the entirety of a library date to around the start of the thirteenth century - before that, memory and ad hoc organizational practices dominated.
It wasn't until the later Middle Ages that shelfmark systems began to be employed with regularity, usually involving a reference for the cabinet or unit, another for the shelf, and a third for the place on the shelf, e.g. K-K 1 or something to that effect. The famous Cottonian system cited in this thread was a variation on this system.
There is plenty more to discuss as well: the chained libraries of the mendicant orders, for instance, are another way of organizing libraries. But that should serve as a working answer for the European Middle Ages, at least.
Medieval libraries had typically the following convention:
Four volume Bibliotheca universalis (1545–1549 ) by Conrad Gesner was the first comprehensive attempt of listing all books, It listed over 10,000 books alphabetically. He also wrote companion for Bibliotheca universalis, Pandectarum sive Partitionum universalium Conradi Gesneri. It contained 21 main categories and many subcategories.
De grammatica et philologia
De dialectica
De rhetorica
De poetica
De arithmetica
De geometria
De astrologia
De divinatione cum licita tumillicita et magia
De geographia
Here is nice article that describes how the The Vatican Secret Archive was indexed. Cardinal Giuseppe Garampi An eighteenth-century pioneer in indexing by Charles Burns http://www.theindexer.org/files/22-2/22-2_061.pdf
Paris Bookseller's classification from mid 17'th century was the first modern classification system. I think Biblioteque Nationale and British Library used the system derived from that before Dewey and Library of Congress Subject Headings were adopted.
Source: World Encyclopedia of Library and Information Services, 1993