I was reading about "striking the colors" vs raising the white flag on Wikipedia ( http://en.m.wikipedia.org/wiki/Striking_the_colors ) and the following passage made me wonder how they communicated with such detail while in battle against each other.
"During the same battle, Bonhomme Richard's ensign was shot away. When Pearson saw that the colors were down, he asked Capt.John Paul Jones of Bonhomme Richard if Jones had struck his colors. Jones has been quoted as replying, "I have not yet begun to fight"."
How did he ask Capt. John Paul Jones if he surrendered? How did the captain reply with such a detailed sentence?
Actually, in that case they literally yelled at one another. Ship fighting ranges were very close at the time, and captains and officers had very developed voices (and could use megaphones, etc.)
As to your larger question, the British navy had very developed systems of signaling with flags, which waxed and waned depending on the fleets and admirals in charge. Both general and specific directions could be made with combinations of flags, and an alphanumeric system could spell out anything not in the book. At night, ships could signal with lanterns or gunfire; bells, guns or even horns or conch shells would be used in a fog.
Then signalling by morse code using an Aldis light, or by Semaphore using smaller flags became common as well. Signal flags, in many combinations, were still in use until at least the 1990's. If you see a ship arriving in port today they should still be flying a yellow flag (Q) to show that they are "healthy", and a red and white flag (H) to show that they have a pilot on board.