If it was good enough for the Ancient Greeks ("no, there is absolutely no military threat being presented by this humble gift of a large wooden horse"), it was probably good enough for anyone well read on the classics in the 18th Century.

What you are looking for is the term Ruse de Guerre, or 'ruse of war' in English. Making your ship look entirely harmless is not uncommon at all, either by pretending to be an unarmed merchantman (Q-Ships), or pretending to be another country entirely. For example, no major German vessels in WW2 had multiple front turrets. Here is Graf Spee with a fake second one up front, to give more time to get closer.

It can go as far as flying the enemy's flag. The general rule is that you're not allowed shoot when pretending to be the enemy, but using enemy insignia to get close enough is normally allowed. A more extravagant case in point was HMS Campbletown, which not only was visually modified to look German, but flew a German flag in order to get closer. It swapped out for the British ensign once it was obvious that the jig was up and they had been identified as a threat, then they shot back.

A land-based version would have been the use of German soldiers in US uniform during the Battle of the Bulge. Though some of the Germans captured were tried by tribunal and executed at the time, in the trials after the war, the conclusion was that there was no evidence that the German forces had been operating as spies (i.e. to return information to German lines) or instructed to fire upon Allied forces when in US uniform).

Forgive my focusing on the 20th Century examples, the Age of Sail is not my area of specialty, however, the written rules on the subject would not have been notably different. Well, other than the fact that there were fewer of them, the Hague Conventions were a 19th Century development.