I recognize that the answer could vary wildly depending on the political/economic climate of any given region, but the thought occurred to me when I was playing through Elder Scrolls Oblivion and ran into a bandit compound based in an abandoned fort, almost directly outside the main gate to the Imperial City. It seems like nothing short of supreme negligence would allow that to happen, but I also wouldn't be surprised if, at certain tumultuous times in history, a city or kingdom found itself almost completely overrun with robbers and bandits.
Banditry became a real problem in England during the reign of Edward II in the early 14th Century, especially following his confrontations with his barons and defeats in Scotland at the hand of Robert the Bruce, such as at Bannockburn in 1314. It would continue to be a crisis for much of his reign. The real causes of this are political. Edward II was embroiled in conflicts with his barons over his relationship with a courtier, Piers Gaveston and this essentially boiled over into a civil war in 1312. In 1314, defeat at Bannockburn also drove the English effectively out of Scotland, and continued political disagreements led to a second baronial uprising in 1321, before Edward's own wife and the exile Roger Mortimer led an invasion which deposed Edward II in 1327.
The result of all of this political upheaval and internal conflict was essentially anarchy. In the North, with the army and the marcher lords in disarray following Bannockburn, the Scottish were able to launch retaliatory raids into Northumbria. At the same time, the series of successive conflicts had led not only to a famine, but also large numbers of desperate, disposessed and heavily armed men left to roam the countryside in the wake of baronial and royal armies. Highway robbery became a massive crisis: Edward II was unable to guarantee the safety even of members of his own household or prominent ecclesiastical visitors.
Royal attempts to militarily oppose the bandits were hamstrung by a lack of available soldiers, and the crown often had to fall back on ransoming its own household, judges, sheriffs and even cardinals. In the 1320s, royal troops defeated a bandit group near Stafford, but on returning to the town found that the gates had been barricaded by bandits and they were unable to enter. In the 1330s in the wake of Edward's reign, the situation if anything got even worse, with Parliament petitioned many times to deal with rampant kidnapping of prominent officials and clergy.
Some respite was brought by a combination of The Black Death and the Hundred Years War. The war effectively exported banditry en masse to France, where the vast chaos caused by the conflict allowed for rampant banditry on the part of the English army. The devastation caused in France by banditry was truly remarkable. The Italian scholar Petrarch, who visited France in 1360, remarked that "the most oppulent of kingdoms is a pile of ashes." By the 1440s, it was remarked that "where there had once been fine manors, domains and heritages, towering bushes grow."
Some reading:
Bellamy, Crime and Public Order
Shlapentokh, Societal Breakdown and the Rise of the Early Modern State in Europe
King, "Schavaldours, Robbers and Bandits: war and disorder in Northumberland in the reign of Edward II" in Thirteenth century England: 9: Proceedings of the Durham Conference, 2001
Neveux, "Les marginaux et le clergé dans la ville et le diocèse de Bayeux au XIVe et XVe siècles" in Cahier des Annales de Normandie 13
Hi, not discouraging further contributions here, but meanwhile you may be interested in a few previous threads on this subject
How much truth is there to the generic medieval bandit depicted so often in media about or based on the middle ages? featuring /u/idjet
How prevalent was the threat of bandits in medieval Europe? Is it somewhat exaggerated in popular misconceptions or were there really bandits lurking all over to prey on unwary and unguarded travelers? featuring /u/NewYorkeroutoftown
Medieval and "Fantasy" stories often use the trope of a bandit attack on a village. Was this a common thing anywhere in the world? featuring /u/crackdtoothgrin
How safe was medieval Europe? featuring /u/sunxiaohu