There was no universal system for hiring or using "guards" in the medieval period. Different problems often meant different solutions, and that might mean several overlapping solutions operating in the same city.
First let's talk about "guards." Guards is a lousy word to use, to be honest. It really doesn't have much of a functional parallel in medieval or early modern cities, not in the sense that we mean it in Dungeons and Dragons or other pop-culture takes on pseudo-medieval settings that imply a citywide system, like modern police forces. Instead, medieval cities had a number of different entities with different and very limited roles. There were private guards, which could be hired by individuals to either protect them in market places or on the roads - bodyguards, caravan guards, estate guards, etc - and then the city would likely have some form of standing or semi-permanent force to mind the gates - gate guards.
Given the role of a specific person or location to protect, "guard" works. It's a relatively passive, or at least reactive, job. Open and close the gates at a certain time, give the alarm when necessary. Subdue aggressors. Andre Paurnfeindt, a 16th century Trabant for Cardinal Matthaus von Wellenburg, actually wrote a fencing treatise, and among some of his unique contributions to the corpus of Fechtbucher (fencing books) was his inclusion of a few plays that included grappling on the ground, or holding your opponent on the ground:
Throw them down on their stomach, thus fall on them with your right knee on their back, and with the left hand grasp a tuft of their hair, and tug upward, twisting their neck, so they fall again on their stomach, if they try to stand.
Sounds a little familiar, huh?
But cities need more than protection for the wealthy and minders at the gates. This is where things get pretty complicated, because medieval and early modern cities had specific roles carved out for specific tasks that relate to what we would consider the role of law enforcement. In 1583, a man named William Lambarde wrote a treatise covering the role of these diverse agents of the peace, titled:
The duties of constables, borsholders, tithingmen, and such other low ministers of the peace whereunto be also adjoined the several offices of church-wardens, of surveyers for amending the highways, of distributors of the provision for noisome fowl and vermin, of the collectors, overseers, and governors of the poor, and of the wardens and collectors for the houses of correction
Which gives a pretty good overview of the number of officers in a city, and their limited roles. Taken all together, these positions would certainly give a similar kind of coverage of modern police forces, but individually, each of these positions has a very limited role. The most important position here, though, is the constable, a position (usually elected, but not always) that functions kind of like a supervisor of these officers. Added to those duties was also the responsibility of making arrests, and responding to citizen's complaints or warnings (more on that in a second), and working closely in conjunction to the justice of the peace. Arrests, when made, would hold the suspected felon in some secure place within the constable's ward, sometimes this might be a locked room in a tower (near the gate guards), or a secure room elsewhere.
However, Lambarde represents arrests being as much the responsibility of the citizenry as of officers of the peace. Here's a description of what is to be done by highwaymen arrested by civilians:
if one does assault a man, in, or nigh the highway, to rob him, and be taken by the true man, or by any other, and be brought to the constable, or such other officer of the place: then ought such officer, not only to take him to his ward, but also to carry him before a Justice of Peace to cause him to give surety for his Good bearing.
In some places, and especially in the 16th century, citizen militias performed a lot of the humdrum tasks of law enforcement. They were the fire watch as well as the men who walked specific patrol routes, called in emergencies, and were meant to be the first-responders in case of emergency. Each city had its own system or organization for the watch, and it wasn't uncommon for citizens to pass off their responsibility to hirelings they called substitutes. But the notion was that every citizen was himself a member of the law enforcement apparatus of the city. They were legally obligated to interfere in fights, help apprehend criminals and prevent crimes, and respond in an organized way to emergency. The notion of the "hue and cry" the call raised by every able-bodied person who witnessed a crime (especially murder) was a deeply ingrained cultural practice from even before militias were organized.
The constable could, on his own authority, make arrests for suspected persons for murder, theft, arson, or other such high crimes. It was, as Lambarde noted, "a chief part of their office to repress felons." However, constables were also responsible for making arrests "if information given... that a man and a woman be in adulterie, or fornication together... and if he find them so, he may cart them to prison."
The history of law enforcement is full of stuff like this, of poor laws and vagrancy laws and sumptuary laws, that are generally viewed by historians as aspects of what we call "social control theory." The crude version of social control theory is that agents of law enforcement (along with other, more positively-viewed social apparatus) were not actually meant to keep people safe, they were meant to control populations, and make them more amenable to government influence. This is certainly true of a lot of the work done on early policing, where the relationship between municipal police departments and private armed security and slave patrols in the southern United States is made explicitly. Those relationships definitely exist, and while it's (in my opinion) a little deterministic to see it entirely through that lens, it's certainly something we should keep in mind, and it's important to emphasize it here: early police were viewed as having a role in society that is markedly different than the role we ascribe to police today.
It would be remiss not to mention the role of thief-takers and bounty hunters in this capacity. There were a huge number of roles that were taken by private enterprises or individuals to apprehend criminals under the direction or approval of more official law enforcement bodies, and the thief-taker was a big part of this. Infamous theif-takers like Jonathon Wild have likely exaggerated their role somewhat, but they seem to be a combination of criminal informant and bounty hunter, operating with the sanction of the local constables. Wild himself was eventually arrested and convicted of theft, which might tell you something about how this system functioned.
To make a long story short, there were a number of what w would consider public officers of the peace, operating alongside private enterprises, and non-governmental organizations, but even those neat categories aren't quite 1-to-1 in comparison to the systems in place in many places in the medieval and early modern period. To answer your question: do guards function more like police or private security? Both, and neither.
I answered a similar question just the other day, and another one a long time ago
You can find William Lambarde's The Duties of Constables online
Information about early modern cities (especially Germany) come from Tlusty's The Martial Ethic in Early Modern Germany