A lot like how it works today, just slower. Descriptions of the anonymous buying and selling of books goes back at least as far as classical Athens, where Socrates in the Apology describes the possibility of purchasing texts by philosophers like Protogoras outside the theater. I stress anonymous, incidentally, in order to separate books and literature as commissioned work and books and literature as commodity. A commissioned work might be an individual specifically ordering the creation of a given text, whether from the author himself or from a copy (one might think of Ptolemy ordering the copying of Athenian tragedies). Anonymous purchasing would be a buyer acquiring a copy that was not made for them specifically, from a seller who had the intention of selling to any particular individual. This is a little bit pedantic I grant, but I think it is important to stress the social aspect of this.

Anyway, there are a handful of mentions of anonymous book acquisition in pre-Roman times and a whole bunch of them in the Roman period. This included something like a publishing industry--Cicero's friend Atticus, one of those wonderful pre-capitalism capitalist types, reportedly set up a publishing house with scribes (probably slaves, but I can't remember if it is specified) to copy out works for sale. There were also bookstores that were often attached to the copy houses and sold to walk in customers. There are several descriptions of bookshops in the Roman world, often by poets who were rather put out by the booksellers were making all the money selling the books they wrote. There was also one writer, unfortunately I can't remember his name, who wrote about dropping in to bookshops to see if his work was being sold.

The big differences were on the front end of literary production--there wasn't much in the way of copyright law, so making money directly from book sales was not feasible. It was possible to make a living writer, but it was only through patronage or prize competitions. Many were also independently wealthy, as literary skill and production were a mark of prestige for the Roman elite.

Peter White has an excellent essay "Bookshops in the Literary Culture of Rome" in Ancient Lteracies, but if you can't get to it Mary Beard wrote a nice little column about the topic.