"Before GPS" is a pretty large chunk of history, :-) but if you mean mostly in the Greek/Roman era I can attempt an answer.

First off, most naval battles throughout history have been fought near straits, capes, harbors or other obvious choke points where ships could wait for an enemy fleet, or, conversely, where an aggressor fleet would be forced to fight its way past a fleet to reach an objective. (The only major battle of the Age of Sail that springs to mind which was fought far offshore was the Glorious First of June in 1794 -- the Wikipedia description is perfectly adequate.)

The tradition of near-shore battles really didn't change significantly until the 20th century, when scouting fleets or scouting aircraft, and later radar, could find fleets farther offshore. Jutland, for example, was fought in the middle of the North Sea based on British decryption of German signals that indicated the High Seas Fleet was leaving port; battlecruiser units of the two fleets eventually sighted one another and started the battle. But I digress ...

Anyhow, in ancient times, it's long been thought that mariners navigated by hugging the coast (staying in sight of it at all times). I've always been a bit skeptical of that (as have other naval historians) because the coast is where the reefs and rocks and lee shores are, and a lot of sensible captains would want to stay out further from that. We also know that not all ships of the era were oar-powered, so merchant or sailing ships would have even more reason to avoid coasts if possible. The current thought is that in Europe and in days before the compass and longstaff/quadrant/sextant, most navigation was carried out by visual cues; east and west are fairly easy to figure out at sunrise and sunset, and north is indicated by Polaris at night. Waves and currents behave differently in the presence of land than farther out at sea, which can also give a navigator clues as to where they are, and landfall looks different depending on where you are. Moving into the area of speculation, we assume that ancient mariners would use something like the rutters or sailing books we see before the invention of charts, which are basically sets of written or oral descriptions of how to get from place to place. We don't know that for sure, but it seems reasonable that that knowledge would get passed along.

Anyhow, I rambled a bit, but I hope that helps!