The notion of "frontier" could mean different things for pre-modern people. For the Romans, the "Frontier" was that point beyond which the Roman way of life had yet to arrive. For the Greeks, the "Frontier" was that place beyond which the greek language was not understood or widely spoken.

But if you are asking how borders worked prior to our modern nation-states, the answer is that they largely didn't. I actually wrote about how borders often worked in the past in this answer from awhile ago.

Even with a river or mountain marking a border, sealing off contact between two neighboring communities is largely an invention of the industrial era. Even when the vast majority of people were involved in agriculture, people in towns and hamlets interacted in order to trade goods and services, creating a largely uninterrupted cultural and linguistic continuum. In neighboring or nearby communities along this continuum, social identity or economic role might play a larger role than geographic identity in determining who someone was and how they were able to transit.

Monarchs and aristocrats might draw lines across where their jurisdictions and rights to collect taxes ended, but man-made barriers existed more for toll collection at places like city gates, or bridges and mountain passes. The idea was to create economic barriers and extract taxes, and not really to restrict and control the passage of people, as modern border controls do. In the past, the concept of control and regulation of people based on their identity was instead entirely relative, often the result of specific political tensions coming to a head, and not the result of drawn-out policies like the ones that have emerged in the industrial era (and later, in the information age).