A compass is a navigation tool that tells you the direction you are sailing in. Other methods of determining direction of sailing are:
Using the sun to determine the direction. It's easiest to use the position of the setting or rising sun along the horizon, or the sun at zenith (i.e., local solar noon), but can be done at any time of the day with a suitable sundial. This depends on having a sufficiently clear sky (or the use of polarised light, which has been suggested (rather speculatively) as a possible Viking technology - see http://dx.doi.org/10.1098/rstb.2010.0194 for details (which also includes a description of sundial navigation)). Doesn't work at night.
Using stars to determine direction. Needs clear skies, and doesn't work during the day.
Using the direction of consistent known prevailing winds to determine the direction of sailing. Needs knowledge of local winds, and is less reliable that using the sun or stars, since the direction of the wind can vary. Can be useless at times of the year where the wind doesn't consistently blow in one direction. Can be used day or night, and in poor visibility.
Use the direction of prevailing swells. Needs knowledge of local wave conditions. Less reliable than sun and stars, more reliable than wind.
Use landmarks or seamarks. Needs local knowledge, and only works within sight of the landmarks or seamarks.
Note that local knowledge can be obtained by using a local pilot.
The accuracy of navigation by sun and stars can be improved by using instruments (e.g., sextant, but a wide range of instruments has been used), but such navigation can be done by eye alone.
There are methods of navigation that use methods other than determining the direction of sailing. For example, methods that indicate the presence of nearby land (while it is still out of sight). These methods have the advantage of being useful even if you are sailing in unknown waters, without maps/charts. Methods of finding nearby land include:
Types of birds in flight (land birds can indicate nearby land) and their direction of flight.
Clouds can indicate land.
Changes in wave direction. The pattern of waves in the ocean is affected by islands; this can be used to detect islands.
Changes in water depth (which can be measured by sounding with a lead or pole).
Changes in water due to outflow from a major river.
Other than these, the simple method of sailing within sight of the coast was commonly used.
Seán McGrail, Early Ships and Seafaring: Water Transport Within Europe, Pen and Sword, 2015, has a concise discussion of navigation methods, including most of the above methods.