There are lots of great books about Port Royal in its heyday. The first ones I'd recommend are The Sack of Panama by Peter Earle and Empire of Blue Water by Stephen Talty which both give really solid broad introductions to the politics and environment of the Caribbean and Port Royal's relationship with buccaneers during its heyday in the 1660s until 1671 when England started to crack down on them.

If you want more specialized reading exclusively on Port Royal then I'd recommend Pirate Port: The story of the sunken city of Port Royal by Robert F. Marx for some light reading, and if you want a really excessively meticulous study of everything you ever wanted to know about Port Royal from written records and archaeological findings with lots of maps and reconstructions included then read Port Royal Jamaica by Michael Pawson and David Buisseret.

Lastly, a great primary source on Port Royal in its heyday is the contemporary book The Buccaneers of America which was published by Alexandre Exquemelin in 1678. Exquemelin himself was an actual former French/Dutch buccaneer and the book contains many of his first-person recollections, such as this describing the activities of buccaneers in Port Royal in the 1660s which has clearly influenced some modern pirate tropes:

Captain Rock sailed for Jamaica with his prize, and lorded it there with his mates until all was gone. For that is the way with these buccaneers -- whenever they have got hold of something, they don't keep it for long. They are busy dicing, whoring and drinking so long as they have anything to spend. Some of them will get through a good two or three thousand pieces of eight in a day -- and next day not have a shirt to their back. I have seen a man in Jamaica give 500 pieces of eight to a whore, just to see her naked. Yes, and many other impieties.

My own master used to buy a butt of wine and set in the middle of the street with the barrel-head knocked in, and stand barring the way. Every passer-by had to drink with him, or he'd have shot them dead with a gun he kept handy. Once he bought a cask of butter and threw the stuff at everyone who came by, bedaubing their clothes or their head, wherever he best could reach.

The buccaneers are generous to their comrades: if a man has nothing, the others will come to his help. The tavern-keepers let them have a good deal of credit, but in Jamaica one ought not to put much trust in these people, for often they will sell you for debt, a thing I have seen happen many a time. Even the man I have just been speaking about, the one who gave the whore so much money to see her naked, and at that time had a good 3,000 pieces of eight -- three months later he was sold for his debts, by a man in whose house he had spent most of his money.

Hello - The starting place for seventeenth century Caribbean history is still Richard Dunn, Rise of the Planter Class in the English West Indies. Dunn is not specifically interested in pirates, but he deals with them as the need arises, and covers the career of Henry Morgan (pirate and Royal Governor).

The standard work on the history of Port Royal itself is Pawson and Buisseret, Port Royal, Jamaica (1975). Buisseret also edited and published the sections of John Taylor's unpublished history, which includes a detailed description of Jamaica in 1687 (Buisseret, Jamaica in 1687: The Taylor Manuscript). Another primary work worth consulting is Exquemelin's The Buccaneers of America - it is one of the main historical references on piracy in the seventeenth century. Lane, Pillaging the Empire: Piracy in the Americas is an excellent introduction, and worth consulting for it's references if not for the main text itself. David Cordingly's works, especially Under the Black Flag are good at comparing the mythology and reality of piracy.