This is a difficult question to answer as ‘enforcement’ can be a relatively ambiguous concept. I will answer based on my own field of expertise, taking the Byzantine-Arab ‘frontline’ as an example (roughly from the seventh to tenth centuries). The reason I have used inverted commas for frontline is because the frontier was in fact more of a ‘zone’, which itself was a fairly fluid concept, waxing and waning across the Anatolian plateau over the decades. This has been sometimes referred to as the ‘deep frontier’ by scholars (see, for instance, the works of F. Trombley or A. Eger cited below). I think especially Eger’s work is very relevant in context of your question.

Numerous Byzantine sources from this period show that the border was fairly porous with incursions happening (sometimes even annual raids) across the frontier zone for purposes such as pillaging and captive-taking (sources such as Theophanes, Agapios, Michael the Syrian, al-Tabari, the Peri Paradromis, and hagiographic work - all listed below in detail). 

Despite the ‘porous’ and relatively fluid format of this frontier zone, in times of increased stability on the Byzantine side (following the 863 Battle of Poson for instance) it did become more of a narrower ‘frontline’. This frontline stretched across the Taurus and anti-Taurus mountain ranges thus taking advantage of the geography of Asia Minor. So, one answer to your question of ‘how was the frontline maintained’ is through taking advantage of geography. These mountain chains which separate the Anatolian Plateau from Cilicia feature relatively few passage-ways (mountain passes). This makes them both easier to defend, and also to observe.

In the Byzantine period there was a beacon-lighting system. This is attested to most directly in the tenth-century De Ceremoniis text (De Ceremoniis, Book 1, Appendix 1.3). It mentions a series of fortresses with visibility of each other (or at least visibility of their fire/smoke signals) extending from roughly the frontline to Constantinople. This would give advance warning of enemy movement. This was one means by how preparations could be made quickly to confront enemy army columns. An anonymous Byzantine military treatise dating to the tenth century mentions signal fires being utilized to warn of imminent enemy attacks for the surrounding villagers and townspeople, so that they fled towards fortified sites for shelter (this is the Peri Paradromis). Hagiographic sources also frequently allude to the phenomenon of villagers seeking shelter behind fortified strongholds during times of hostility in the provinces (such as the St. Theodore material for Euchaita). Thus signalling had a dual function of both alerting the respective localities and inhabitants of the provinces to take shelter, while also helping convey the distress signal to decision-making centers. More information on beacon-signalling can be found in; S. Mitchell, Anatolia (1993); and regional surveys also imply that this system existed elsewhere (e.g. the Pontus - see E. Sökmen, Surveying the Pontic Landscape through Fortresses (2016)).

That being said, Byzantine military manuals underline that it was often easier to let the Arab armies through the passes, allow them to ravage Byzantine territory, and then ambush them on the way back out through the mountain passages, as they would at that point be tired and weighed down by acquired plunder (Peri Paradromis, chapter 23). One example, narrated by Theophanes in his Chronographia (AM 6263) for the years 771-772 featured the Byzantine army calculating the Arab army’s exit trajectory through ‘shadowing’ the army (scouting) and occupying the relevant mountain pass in preparation of ambush. Another example is the ambush which Leo Phokas orchestrated in a mountain-pass in the Taurus mountains while an Arab army under the emirate of Aleppo was retreating in the year 950 (see Treadgold, History of the Byzantine State, p. 489 for a discussion). Thus, this is a somewhat indirect answer to your question on how the ‘frontline’ was enforced. While not necessarily enforcing the frontline for entry, this shows that the Byzantine defense often involved ambushing on the exit. This is also tied into the broader arguments of utilizing the geography of the region (sparsity of passage-ways) to help ‘police’ the border in a sense.

The porousness of the ‘border’ (in contrast to its modern equivalent) is also evident in what A. Eger highlights through the vibrant cross-border exchange (frontier exchange) which occurred over it, between the Muslim and Christian sides. Thus non-military movements also occurred abundantly and in an unregulated fashion across the so-called frontline, showing it was not a true border in the modern sense. For instance, the pastoralist products (dairy and animal products) of the Byzantine side in Cappadocia would be exchanged for grain from the Islamic side, and this is described as ‘business as usual’ by the general Nikephoros Ouranos (Eger, p. 262)

I have focused on a specific case of a Medieval ‘border’ to use the term you initially posed in your question. I hope I have managed to illustrate how complex the whole subject is and I would strongly urge anyone interested in the subject (as it pertains to the Byzantine case) to go through the bibliography below, and particularly the works of Asa Eger and John Haldon.

Primary Sources:

-Theophanes the Confessor (Chronographia).

-Michael the Syrian (Chronique de Michel le Syrien, patriarche jacobite d'Antioche, 1166-1199, éditée et traduite J. B. Chabot, Vol II (Paris: E. Leroux, 1901))

-The History of al-Tabari (The History of Al-Tabari Vol. XXIII: The Zenith of the Marwanid House, The Last Years of Abd al-Malik and the Caliphate of al-Walid A.D. 700-715 / A.H. 81-96, trans. Martin Hinds (1990)).

-Agapios (Kitab al-‘Unvan: Histoire Universelle écrite par Agapis (Mahboub) de Menbidj (Vol. II), éditée et traduite A. A. Vasiliev, in Patrologia Orientalis 8 (1912))

-Hagiographic Work (Haldon, Tale of Two Saints for St. Theodore material; The Vita of St. Philaretos (ed. Fourmy); The Vita of George of Amastris (ed. Vasilievskij))

-The Peri Paradromis or “Treatise on Skirmishing” (found in Three Byzantine Treatises, ed. G. T. Dennis)).

-De Ceremoniis (Ann Moffatt, Constantine Porphyrogennetos: The Book of Ceremonies, Byzantine Australiensia Vol. 18 (Leiden: Brill) (2012)).

Some Secondary Sources on the Subject:

-A. Eger, The Islamic-Byzantine Frontier: Interaction and Exchange Among Muslim and Christian Communities (2014).

-J. Haldon, Byzantine Warfare (2017)

-J. Haldon, The Paradox of Eastern Roman Survival 640-740 (2016)

-W. Treadgold, The Byzantine Revival 780 - 842 (1989)

-N. Roberts et. al., “Not the end of the world? Post-Classical decline and recovery in rural Anatolia”, Human Ecology 2018 (3).

-F. Trombley, “War, Society and Popular Religion in Byzantine Anatolia (6th-13th Centuries)” in Byzantine Asia Minor 6th – 12th Centuries (1998).