The masts of a tall ship are, well, tall; the main mast of HMS Victory (when all its parts are attached) reaches 220 feet above sea level. Even if an intrepid sailor isn't actually going to stand on the top of the mast, a view from 200' gives you a horizon of about 17 miles or so, and assuming you're sailing in a fleet with other ships overlapping your line of sight, an admiral could sweep a surprising amount of sea in the course of a day. Keep in mind also that you can see the masts and sails of another ship when it might be hull down over the horizon, due to the curvature of the earth. And sailing ships in the period I study mostly stuck to predictable routes, and/or had areas they needed to pass through to get from point A to B (there's a reason the British are in Gibraltar still).

I wrote about Nelson's tactics at Trafalgar here before, in a post that explains how his ships caught up with then lost sight of the allied fleet when they were chasing it to the Caribbean and back. I wrote about naval blockades here, in a post that includes info about the process of keeping ships bottled up in harbor and looking for them when they broke out.

Presuming you mean pre steam. Numerous sources, notably campaign accounts give various answers to this e.g.

• You generally (hopefully) had at least some idea of the enemy's rough location, intentions etc. With you additionally using every possible source of information e.g. questioning passing shipping, popping in to ports and harbors along the way etc. - deception re this an important military tool (particularly with such short range ships as galleys).
• All ships and particularly larger forces constrained by the need for provisions - food, fuel, water etc. and usually had to stay close to points where they could replenish or at least top up these. Limiting the places you had to look.
• In the days before good navigation and accurate charts commonplace, (let alone sat nav - which they would have happily killed for). Most shipping generally stuck closely to familiar trade routes and usually coastal waters as far as possible.
• Standard practice was to put out a screen / line of (usually minor) ships to maximize your sight horizon. But in the days of sail this hampered by wind and weather and a decent signalling method important - the British Nelsonic period flag and other system an example of the best of its time (but other previous systems must have been reasonably effective and sophisticated - but less well recorded). At base you got close enough to physically speak to the other ship or send them a boat.
• The seas are BIG and blue waters interceptions with the available methods rare. So to intercept. Your best chance was to loiter around their objective / the place you were defending.