The short answer is that they couldn't necessarily tell the difference - at least not until it was too late.

All ships at the time (and today, actually) were required to fly flags indicating country of origin and what kind of ship they were. Flags can also be used to signal other things like "man over board" or that the ship is in distress and needs help.

So when pirates ships were sailing out on the hunt, they would keep a chest full of different flags for different occasions. Coming up on a Spanish merchant ship? Run up the Spanish flag! They'll think we're friendly and let us come up along side without a struggle. Comming up on an English man of war? Fly the colors of a country the English aren't at war with. This flexibility in flags allowed then to get close to possible prey ships, and avoid ships that they might get in trouble with. The famous pirate flag was run up as a tactical manouver. You used it to intimidate your prey as you approach, or just before you attack. But they generally didn't sail around flying the skull and crossbones - it was too dangerous.

Because of this practice, "flying false colors" (flying the wrong flag) was one of the most common charges laid against a pirate when he was unfortunate enough to end up in front of a magistrate.

Depending on the particular pirate crew, it might also have been possible to determine the difference between a pirate and a naval vessel by how well maintained the ship was. Keeping a hunk of wood and a bunch of metal, fabric, and hemp from disintegrating in the salt water was a battle that had to be fought daily, but some pirate crews decided to forgo the daily routine of deck scrubbing, line checking, barnacle scraping, and so on in favor of more satisfying pastimes. Therefore a ship that looks like it's being held together by bubblegum and duct tape is more likely to be a pirate ship than a naval vessel.

To be fair though, everyone used to avoid the navy if they could. Pirates had the obvious reason, but if the navy boarded another ship from their own country, they could "press" sailors from that ship, meaning they would force those sailors into the navy. It's bad enough if they take anybody off the crew since merchants at the time we're much like merchants today - they wanted to employ as few people as possible to get the job done. Hence, most ships, like most businesses today, were run by a crew that was just too small for the task at hand. So the loss of any of them would be a problem. However if the navy ship took your only navigator or your only carpenter (which they would do if they need someone with those skills), you might literally die. So ships of all kinds steered clear of the navy if possible, especially if it looked like they may have recently been in a battle, which might mean they'd be looking to replace a few fallen soldiers.