Sorry for the late response, but you can check out my old responses:
Basically, the ships would approach each other quite close (usually every evening) and then converse freely or at worst yell to each other (exceptionally they could send written messages by small boats)
There was also rudimentary signalling used for basic messages when the ships were further apart: rasing of the flag/banner (or even sails), lighting lanterns in the night (visual signal) and firing guns (audio) which would usually be predetermined signals what do they mean