On the level of the individual Squadron or Fleet you would have signal flags. Which are still very much in use in every navy and every civilian vessel afloat. You could spell individual words or some flags had complex meanings such as "divers out" or "maneuvering impaired". And navies would have specific codes and all such as "attack corresponding enemy ship" and signals could be relayed to ships further from the flagship.

An admiral would also typically explain his plan to his captain's and subordinates before a battle and would also have standing fighting instructions to govern tactics. You also could have signal lights which could transmit in Morse code.

And when not actually fighting there were the options of simply taking one of the ship's boats to visit and discuss in person. Or an officer with a suitable command voice and speaking trumpet is no joke and could be heard at several hundred yards.

Traveling now but will come back and expand on how sailing navies operated on the global and strategic level more later. Unless another fine flair or user can in the meantime!

/u/DBHT14 is absolutely correct but I'd like to add on a little more detail.

Ships most often communicated amongst each other with signal flags, but these were not just confined to grouped or massed ships. The French and Royal Navies, for example, developed signal books for specific theaters of action or even for the entire globe. Within these books, common words or phrases would be attached to certain flags and words not included would be spelled out letter by letter (as /u/DBHT14 noted). A vessel's midshipmen were responsible for knowing the contents of the flag book, knowing how to spell out words, and how to operate the signaling halyards. Competent captains drilled these boys on this skill relentlessly, as seconds counted in battle or storms.

The most important aspect of such signal books were their secrecy. The merchant fleet generally utilized a standard book across nationalities and even across nations. It generally remained static, as in it rarely changed. Naval signal books on the other hand changed frequently. If a ship is captured by an enemy and its signal book is found, the enemy ship essentially has the ability to read your diary.

Changing the signal books frequently was a method to prevent an enemy from utilizing friendly signals to confuse fleets and/or trick individual ships into becoming easy prey through a ruse de guerre. Generally sailing ships looked pretty similar when one ship spotted another across distance. Further, though many ships were uniquely designed, some ships shared similar characteristics and were thus difficult to distinguish (further complicated by the capture of ships, where a ship built by one nation could find itself in the service of another). The best way to tell friend from foe across distance was though the standard flown (national flag) and the signals it flew. But if an enemy ship flies a friendly standard and then utilizes friendly signal flags, it could maneuver close to a foe's vessel without raising alarm - potentially catching the other by complete surprise or at least without it cleared for action.

Naval departments would issue new signal books when it learned of the capture of its currently used book. When warships in port at the time of the change put out to sea, they would carry new books to be delivered to friendly ships they came across or to be disseminated by the commander of a fleet or squadron on station somewhere. In individual ships, all of a captains secret documents (orders, logbook, and signal book) were always kept in a locked desk alongside weighted envelops. In the event that capture was inevitable, it was the captain's duty (or his steward's duty) to toss these precious documents into the sea to prevent capture. Obviously, keeping the signal book secret was a big deal.

Signal flags were also useful over longer distances as well. Standing at sea level on a clear day, an average person can see about 3 miles. From the top of a ship's mast, that distance can be doubled. A "screen" of ships could travel on parallel courses spaced at maximum visible distance and transmit messages down the line to each successive ship. This was often used to comb an area in search of enemy ships as much more water could be covered and assistance could be called upon when a foe was spotted.

Over vast distances, communication was much slower. When ships were not assigned to a squadron or fleet (or were dispatched on separate missions from that body), they would check into naval stations (essentially port towns with governing officials and naval stores) situated throughout the area of operation whenever possible. Orders were sent to these naval stations via packet ship (small, light, fast vessels specifically designed to transport special individuals, documents, and mail) or via warship heading to the area. When new orders arrived on a station, they were given out to the ships that checked in there. Specific messages were often duplicated and sent several times to ensure delivery and rapidity, general messages (plus general news and gossip) would be told to every ship, naval or merchant, who transmitted the message to every ship they came in contact with. Further, the governors and commanders of these stations had a huge grant of authority to make decisions in the absence of orders from the naval department or in reaction to changing conditions of the area.

As above mentioned, general standing orders were common. In a fleet or squadron making changes to the general orders or when planning a specific action, the captains would be called to the ship where the commanding officer was based to discuss, plan, and disseminate orders. When individual ships came in contact with other individuals, they would again communicate by flag at a distance, close the distance once terms were discussed (applies to friends and sometimes enemies) and then a boat would be sent over for captains to meet while the hands of the ship and visiting boat would exchange gossip.

I hope these comments added a little something!