"Policing" as we understand it was the responsibility of every member of the community to greater and lesser degrees. The basic legal framework even in the early 19c. was based on the Statue of Westminster (1285). This document established: night watchmen, hurrying to the hue and cry, serving as parish constables, and generally being sufficiently armed to conduct these roles.

Night watchmen are what you would expect. They patrolled the streets at night to discourage crime. If they witnessed a crime, they were expected to raise the hue and cry. It was expected that all citizens would take turns acting as the night watchman. Like many of the systems discussed here, it worked fine enough in rural communities, but fell apart during the population explosions seen in urban centres during the industrial revolution. Once the role got professionalized, it was also subject to corruption.

The "hue and cry" system: ever see someone on TV shout "stop, thief!"? That was the hue and cry. Under this system a person who witnessed a felony was obliged to "raise the hue and cry", and alert their neighbors to the felonious action. The neighbors were thereafter obliged to assist in apprehending the alleged wrongdoer. If everyone did their part, this system worked well enough in small communities. It was not terribly effective in larger centres, especially after the 19c population growth.

The parish constables were unpaid and part time law enforcement officers. They could be appointed against their will, and the position was generally considered a burden due to the extra responsibilities (but no extra compensation). This left the role open to corruption, especially in larger centres.

So, why did the English cling to this system if it was obviously outdated?

Well, the English were notoriously anti-police and viewed the institution of professional full-time police as inherently tyrannical. When discussing police, the English mind generally turned first to the police founded in France after the revolution and active during the Terror. So, even though the system was failing (especially in large urban areas) they were reluctant to institute a police force. The fact that the only police forces the English had implemented at that point were primarily focused on maintaining English dominance abroad (in India, Ireland, and Canada) didn't help this perception.

A compromise, and stepping stone to real policing, was the Thief Takers. These persons were akin to bounty hunters, but were generally paid by the victim of theft and not a bail bondsman. Anyone could collect a bounty, and the Thief Takers ranged from the earnest Baker trying to earn a little extra money, to individuals who essentially ran double protection rackets in association with a network of thieves.

The latter set up extensive crime networks that operated as follows: If the goods were stolen by a member of the network, the Thief Taker could "negotiate" a release of the property, for a price. If the goods were stolen by a Thief outside of the network, they could use the threat of arrest (and likely execution) to force that person into their criminal organization. Thereafter, the thief would need to pay a cut of all their takes to the Thief Taker to maintain his protection. In the 1700's Jonathan Wild, the Thief Taker General, was essentially the mafia don of urban London.

After Wild was found out, other Thief Taker Generals took his place. Notably Henry Fielding and his half brother John Fielding who established the Bow Street Runners. The Bow Street Runners were, by all accounts, relatively efficient, relatively well trained, and relatively free of corruption. This organization operated as a semi-private/semi-public, semi-official, police organization until the Metropolitan Police were established.

Edit: small additions

I cannot answer specifically regarding the UK; however I can talk about how civil order was enforced in pre-19th Century Italian States (~1500-1796), which can maybe give you a good idea on how things worked elsewhere in Europe.

The army itself was often employed as a way to enforce public order (tax collection, dealing with organized criminal groups, fighting against contraband, subduing rebellions, etc.). Local militias (such as the Venetian "Cernide") were widely used for such purposes, as they were usually tied to the land they were protecting and were less prone to cause trouble with the local populace than mercenaries. Light cavalry units were often used as an effective and rapid way to patrol the land. Such armed groups traveled with a judge, a criminal notary and often an executioner, in order to deal with any captured criminal on site.

Generally, "ordinary" crime was dealt with locally and in a variety of different ways. The population itself often spontaneously mobilized against bandits (the old custom of ringing the church bells to warn the inhabitants about a threat was, in many cases, made compulsory by law). Many States placed bounties on wanted criminals; Venice in particular granted immunity for wanted men if they apprehended another, more important bandit (this method, known as "voci a liberar banditi" basically employed criminals to fight crime).

Local judges could rely on armed men to apprehend criminals and enforce civil order (patrolling the streets, breaking up fights, etc.). These para-military forces could vary greatly in numbers and function, and were informally known as "sbirri" (a term which is still used today to indicate policemen in a derogatory way). Such local groups were autonomous and were not connected with the army. Historian Michael Broers described them as "an organized police corps which was both efficient and powerful". Indeed, when Napoleon tried to implement a common military police force (the "gendarmerie", which would later become the model for many of the later police forces) in the territories he conquered, he often had to deal with the already established local "enforcers".

I'd say that the line between law enforcing and the military was much thinner before organized police forces were established, and that civil order was enforced locally in ways that varied according to necessity, time and place.