This is actually a somewhat simple question, so i will take a lot from two earlier comments of mine, one partially on The Law of the Seas and another on Piracy and Letters of Marque.

Tl;dr: They were mostly hanged, it didn't matter where they were captured. In some earlier doctrines, the nationality of the pirate, victim or the territory of the piracy was relevant, but later on everyone could hang pirates.

Firstly: What are international waters?

International waters are waters outside of state control, specifically, the waters that aren't either internal waters, territorial waters or Exclusive Economic Zones. International waters are also, and historically, known as the high seas.

Originally, Spain and Portugal had sought, with support from the pope, to monopolise the right to trade with colonies and to sail on the open sea (Mare Clausum). The other powers generally opposed this, leading to arguably the most famous legal book ever, Grotius (dutch jurist, 1609) Mare Librum. Grotius argued that the sea was a common good belonging to everyone, that included both the right to navigate and the right to fish.

England however changed opinion once they became a major sea power in the 17th century and, by demand from the king, John Selden wrote Mare Clausum in 1635. This idea did not win general acceptance and Mare Librum eventually became dominant concept, of which England would later become proponents of, during their naval dominance and it fitting well with ideas of free trade.

Concretely, the dominant idea for most of european colonialism was a development of Grotius, writing in 1702, Bynkershoek wrote De Dominio Maris, establishing the principle of state sovereignty of the seas as far as a cannonball could shoot. Outside of this, everything was the high seas, meaning international waters. It's worth noting that ideas of Exclusive economic zones that we have today weren't a thing then before the mid 20th century.

Secondly: What happened to pirates?

Pirates were subject to a concept now called universal jurisdiction. Normally, a country can only put to trial crimes that happen in it's territory or that somehow is related to it's citizens, but under universal jurisdiction, anyone can try that crime even if the crime is totally unrelated to the country. A modern example is the case HR-2010-2057-P – Krigsforbryter in Norway, where a person is on trial for alleged war crimes in Yugoslavia. Universal jurisdiction means it doesn't matter where you catch pirates, where the piracy happened or to which nation the ship or crew belong.

In earlier times, they would not have called it universal jurisdiction, but they often used "enemy of all mankind" (Hostis humani generis). The universal illegality of piracy has long roots even in modern law, for example being deemed illegal by the Third Lateran Council in 1179, but as a dominant legal doctrine it mostly came into fruition during the period you are asking about.

Early on we have Alberto Gentili (a professor at Oxford) in his Commentaries on the Laws of War (1588). He argues that pirates are not a group that can have a legal right to wage war (under the then developing just war theory), this means pirate actions are never just under jus ad bellum, rather they are illegal. For this reason, he dubbed pirates the common enemy of all mankind.

Grotius (1583-1645) , sometimes called the father of modern international law, also sees pirates as outside the normal rules governing and limiting the actions of nations, and thus they can be punished and have their property seized by universal jurisdiction, here from his 1609 Mare Librum:

Wherefore lest different matters be falsely confused, I think a distinction should be made between that jurisdiction which is competent to each in common and that which is competent to each one properly speaking. All peoples or their princes in common can punish pirates and others, who commit delicts on the sea against the law of nations.

There are old court cases illustrating the view of universal punishment for piracy as well, for example Rex v. Dawson (1696) where

The king of England hath not only an empire and sovereignty over the British seas, but an undoubted jurisdiction and power, in concurrency with other princes and states, for the punishment of all piracies and robberies at sea, in the most remote parts of the world.

Universal jurisdiction for piracy was not accepted by all initially. Some jurists advocated a need for some jurisdictional limits, for example it happening in a national territory, or the pirate or victim being a national. Thus, a king could not punish pirates wholly unrelated to his realm. Early english jurists, as per the disagreement on jurisdiction over the sea mention initially, also claimed a right to punish pirates within their part of the sea (as per Mare Clausum). But these views were contested and lost out to universal jurisdiction and the free seas. And during the period you are asking about, some "enemy of all mankind" jurisdiction was the dominant idea.

The typical punishment for pirates was to be killed, often by hanging, an example being the The trial of William Kidd in 1701. He claimed to have a privateering license to legalise his actions, but the court disagreed. Whether they had to tried before being hanged was more unclear, there were some jurists advocating an understanding where they could be hanged without taken to port for a trial, for example the jurist Charles Molloy writing in 1677 that the captain:

may expose them immediately to punishment, by hanging them up at the main Yard end before departure

A last "funny" example illustrating jurisdiction and punishment is the 1705 trial of Thomas Green, sailing in the indian ocean on an English commission against pirates, and was taken and accused of piracy against a scottish ship outside of Calicut, and tried before the High Court of Admiralty of Scotland. The alleged victims were scottish, but the court argued jurisdiction not on that ground, but on the grounds that piracy was a crime against the Laws of Nations. Green and some of his men were then hanged.

Later the alleged victims were found alive and well in India and this alleged "piracy" had never happened, and a lot of the testimony and trial probably just came about by mob pressure and justice.

Lastly, piracy was also an illegal transfer of goods. This meant both that taking the goods from the pirates was not stealing, and that you could be at risk because of having your purchase from a pirate nullified with little chance of getting your money back because the pirate did not have the right to sell it, at least this is what Gentili advocates.

Sources, the parts taken from the other comments might come from the sources mentioned there in addition to these, this is just what i re-consulted when writing this.

Rubin, Alfred P. (1988) The Law of Piracy Newsport: Naval War College Press

Ruud & Ulfstein (2011) Innføring i folkerett 4. utg. (Introduction to public international law 4th ed.) Universitetsforlaget: Oslo

Grotius, Hugo (1609): Mare Librum (The Free Sea) - intr. by David Armitage Liberty Fund

Edit: Cleaned up some mistakes and bad sentences