Calendars are a prehistoric technology for keeping track of time, notably with the goal of tracking seasonal activities — like the movement of game animals, the planting and harvesting of agriculture, and so on. Arguably, if you are keeping track of years, you are in some way dealing with a calendar.

So how do you keep track of years? That's the better question. The fact that there is seasonal regularity is pretty obvious to anyone who is paying attention, much less anyone whose lives depend on mastering food production in each season. Astronomical phenomena have been used to keep track of these regularities since prehistoric times. Some cultures primarily tracked the phases of the moon (and the word for "month" in English still reflects this), while others tracked the motion of the Sun (e.g., with a gnomon — a stick that will let you measure the position of the Sun at its highest point from day to day). Many prehistoric artifacts (like Stonehenge) have alignments that clearly indicate an interest in the solstices and equinoxes, which are "markers" of the change of the year and are easy to track if you care to do so (as someone who finds winter unbearably cold, I definitely track the winter solstice, because that tells me when the days will start getting longer, and is the slow inching towards a warmer climate).

Anyway, that's the basics of how you do it, astronomically. If you're asking how did people refer to years, that varies dramatically by culture. Some did it in the "year 5 of the reign of so-and-so" approach, others did the "year 504 since some mythical start date" approach (which is what we use most commonly in the world today).