National navies like the British and French eventually suppressed most piracy in the Atlantic Ocean. How would a naval ship determine that another ship was a pirate vessel, and not a legitimate merchant? Did they board and search them? What would they look for?
I feel like there’s a few key points which are necessary to discuss without specifically answering your question, as I also have minimal references to draw from. Reality is far removed from what is shown in popular media depictions of the era, after all.
When pirates captured a ship, they often did so with limited resources as merchant crewmen were not eager to fight for their companies. In fact, many of the best known pirates started out on a large canoe fitted with an outrigger and sail called a periagua which could hold 30 men. From there, they upgraded to larger ships. Pirates rarely got involved in large gun battles with other ships. Instead they would fire a shot over the bow of their enemy and the enemy would (usually) surrender. The merchant ships needed their cargo space for actual cargo, so they rarely had cannons (or a lot of them). So if a seemingly merchant ship had guns displayed, a navy could presume it to be a pirate ship.
There were much fewer ships in the Caribbean than what is suggested in movies/video games. Assassins Creed IV would make it seem like there is another man of war every 3 miles or so but it doesn’t really match the reality. It was actually a very rare sight to see one unless it was a slave ship, so if there was a giant ship going the wrong way, the navies might be inclined to investigate it.
ETA: Sam Bellamy captured the Whydah, a man of war and the largest slave ship at the time.
The various countries (mostly) stayed in their own territories if they could. I’m sure you’re aware that British pirates started off their careers as privateers; they were unofficially sanctioned by the British crown to harass French and Spanish ships during the War of Spanish Succession. After the war ended, many former privateers continued to harass Spanish ships. Britain attempted to stop them but Spain was not particularly patient and began using their Guarda Costa to harass any and all British ships found in their waters, whether they were merchant, pirate or naval vessels did not concern them.
The Caribbean has parasites in the water which wrought havoc on wooden boats, to the point that the ships would need to be beached every once in awhile to scrape off the bottoms. It was later discovered this could be mitigated by adding copper to the bottom. At least 1 pirate vessel was discovered while it was undergoing such a procedure.
Pirates were often found by dumb luck rather than investigations! Henry Avery showed up on a massive warship with a flimsy alibi. He managed to escape (presumably) but most of his crewmen were found. Charles vane was left on an island (after a mutiny by rackham) and found by fishermen. Blackbeard arranged a payoff/surrender/retirement to a SC magistrate but a governor from up north surrounded and attacked him. Sam Bellamy got caught in a storm and his boat was destroyed