As you might expect, this varied with time and place. Alphabetical order, however, has a long history as a method of organization for works in a library. So far as I am aware, this was an innovation of a fellow named Zenodotus, first director of the library at Alexandria established by Ptolemy II.

Zenodotus obviously got this gig by being quite the scholar. He had, we know, compiled a glossary of rare words, which he organized alphabetically. It seems a logical guess that he opted for this method of organization for the libraries immense holdings (eventually over half a million rolls!) because he had found it so useful in organizing his glossary. Though, this was not alphabetization in the modern sense. Authors were merely grouped together by the first letter of their name. So, for example, one might find Aristotle either before or after Aristophanes, but both would be found before Plato. Tabs were also a helpful feature; they would serve much the same function as titles and author names printed on the spine of modern books. Though, many ancient tabs did not contain titles. The books, or rather scrolls, would indeed have been housed on shelves.

So, to Zenodotus we seem to owe alphabetization, or rather partial alphabetization. Of course, modern libraries are not merely organized alphabetically, but also in terms of genre and content. So, to use our earlier example, it would actually be uncommon for Aristotle and Aristophanes to be found near one another in a modern library. Aristotle is probably over with Plato, Descartes, and Hume in a philosophy section; Aristophanes is probably with Shakespeare in a drama section. This feature too, was present at Alexandria, though it predates that greatest of the Greek libraries. Aristotle, who in his day seems to have amassed the greatest book collection around, is said (by Strabo the geographer) to have "taught the kings in Egypt how to arrange a library." This is quite likely almost literally true; a disciple of Aristotle, Demetrius of Phalerum, was an intimate of the first Ptolemy. In any case, Aristotle had a great many books, it seems as though he organized them by subject matter, and it is possible that it is through him that this method of organization became firmly established.

(On a related note, Aristotle seems to have been a truly prodigious reader. The opening parts of most of his major works consist of something very near a modern literature review, in which he painstakingly works through what his predecessors had said about his subject. The opening books of De Anima is an excellent example of this.)

Finally, the question of who had access to what. Again, this varied. The practice of lending books dates back all the way to the ancient near east, for example. Here, for example, is an amusing and illuminating omen tablet from ancient Uruk:

He who fears Anu, Enlil, and Ea [several deities] will return it [the tablet on which the work was written] to the owners house the same day.

That one needed to threaten divine retribution for not promptly returning a work does not speak well of ancient near eastern borrowing habits! Other libraries were more strict and more ominous:

He who fears Marduk and Sarpanitum will not entrust it to [other] hands. He who entrusts it to [other] hands, may all the Gods who are found in Babylon curse him!

I, for one, am glad that I merely have to pay a small late fee when I forget to return a book rather than be cursed by the gods. In any case, borrowing is an incredibly ancient practice.

Now, who was allowed to access what works does, of course, vary with time and place as well. The ancient near eastern libraries were closer to large, professional, reference libraries than anything else. Aristotle's immense collection was surely for the use of himself, his students, and his fellow researchers. At Alexandria, the main library was located in the palace, and so it seems highly likely that access to it was restricted to scholars in the employ of the Ptolemies. Another, smaller facility was located in a religious sanctuary and contained a far more limited selection of works. This may well have been intended to serve a wider reading public.

I think that is all your questions. If you are interested in learning more about this, I'd highly recommend picking up Libraries in the Ancient World by Lionel Casson (2002, Yale University Press).

Cheers!