The answer is: flags.

Before Nelson's era, like in the 1600s, the English used a simple flag system. Fighting Instructions, 1653. Fighting Instructions was a book of maritime tactics (21 in all), so there are only 2 pages dedicated to flag signaling in it I think. I'm not sure the particulars, but it was something like a red flag meant attack, a blue flag meant retreat or follow me, etc. This worked well enough when fleets were small and you only needed a few simple commands.

Over the next 100 years, they kept adding flags of different shapes and colors for more complex commands, and this got unruly. The British Admiralty then adopted various codes; such as the Popham Code written by Admiral Popham around 1800. This was sort of a visual flag code, and it allowed a fleet commander to spell entire words with flags. So the infamous quote from Nelson at Trafalgar, "England expects that every man will do his duty", it was sent from ship to ship using a combination of flags in a flag code. You can see a visual representation of that phrase here: https://sites.google.com/site/kidicrypt/poham-s Note that even to signal the two letter word "do", you have to raise 3 flags.

Under the Popham code, there were 10 square colored flags and then some additional triangular penants. There were about 1000 "words" and additionally any word could be spelled out letter-by-letter. Various combinations either represented a number, or a letter. There were also codebooks (because the enemy could see your flags, too) and shortcuts; and each ship would have had identical codebooks and notes. These codebooks were sensitive military intelligence.

Also, you're probably aware of the phrase "ship of the line" and perhaps "crossing the T". British doctrine at the time had warships usually going to battle in long lines. That meant that a ship could really only see the ship directly in front of it when a fleet was in a long line. So there was a long time lag between when the first ship ran up a flag signal and it was spotted by the 2nd, who then had to relay it to the third, etc. Like the childhood game of "telephone," except that each ship had to raise somewhere between 3 and 27 flags to communicate a message.

At night they used lanterns hung from the masts or rear of the ship in various shapes to send signals. Later, ships adopted shuttered lights and used those to send code (and eventually Morse code) messages between ships at night.