I can elaborate on this in the context of the Albigensian Crusade (1209-1229) in the first phase of the war from 1209-1213. The crusade took place in Occitania, also known as the Languedoc, and displacement and sieges are prominent aspects of the conflict. It featured French crusaders fighting against Occitan lords who were perceived as resisting the crusade's aim to extirpate the Cathar heresy.

In the Albigensian Crusade, we see a pattern emerge where residents would either flee or stay at a location depending on a variety of circumstances. For heretical dissidents such as the Cathars, they were likely to flee places that were less defensible (low ground, no walls, etc) due for imminent attack and typically fled for stronger fortifications. Places such as Lavaur, Termes or Minerve were some of these destinations. These places were more isolated, well-defended, and would require a strong force to overcome these fortifications. Manpower was a huge issue for the crusading side so fleeing to more defensible locations, even if they were more isolated, disincentivized the crusaders from attacking them. As such, we know that Cathars were fleeing less defensible fortifications for better ones, and staying there even when they become besieged due to how defensible they were.

This brings us to another point, and a factor that influenced where people could go, is that it was common for less defensible settlements or forts to be abandoned in the face of the crusade. The crusade had developed a reputation for annihilating garrisons that resisted them and terror tactics were causing people, heretic and non-heretic, to flee in the wake of the crusade's advance. This also meant that destroying abandoned castra was important after taking an area. Denying spaces for people to flee to when they could not be garrisoned was important to the crusade's mission in extirpating heresy and subjugating the Languedoc. So the decision to leave a place was often determined by how endangered you felt, how defensible the place you were at was, and whether there was even a relatively safe place to go.

What about those who were forced out and didn't have a choice? There is a particular case where the denizens of a town were expelled to leave only a fighting garrison. At the Siege of Penne d'Agenais in 1212, Hugh d'Alfaro stocked the city with food, water, money, and armaments and garrisoned the city with 400 mercenaries after the crusade targetted that castrum. Our sources note that he expelled the inhabitants from "the youngest to the oldest" and that they burned down the bourg, the suburb surrounding the castrum, to deny resources to the enemy. It seems that by expelling the inhabitants, burning the outside housing, and stocking up to the extent that he did, Hugh was making an extremely practical decision to enhance his abilities to resist. He was able to hold out for a time but eventually made an agreement to abandon the castrum so long as he and the mercenaries could go free. The crusaders acceded to this request.

Where did the expelled residents of Penne go? What about the mercenaries and Hugh? It's not made clear but it's plausible they fled to Montauban or Toulouse which were the major places still holding out nearby. It's mentioned that Toulouse, shortly after the siege of Penne, "was at this time unusually full; heretics from Beziers, Carcassonne and Toulouse, with the supporters of heretics and mercenaries, who had by God's judgments lost their lands, had gone into the city..." Toulouse was also one of the most well-defended places and had a large population. This parallels how Lavaur or Minerve accumulated large displaced communities who had fled the crusade's advance.

There was also the chance that they would be forced out into the countryside to fend for themselves. We do observe a phenomenon of Occitanian lords who had lost their lands roaming the countryside with survivors, oftentimes joining up with the mercenary bands who were frequently employed to fight against the crusade. In doing so, they were hoping to reclaim their lands or just eke out a living amid the chaos. They were known as faidits and it's possible Hugh and the mercenaries opted for this route and took in displaced survivors. We know that a number of faidits and their bands joined the King of Aragon in confronting the crusade at the Battle of Muret in 1213.

TL;DR

The nature of the crusade's conflict incentivized those at risk to flee less defensible places for more defensible ones. These tended to be isolated, mountain fortresses or major urban areas where a siege without a major attacking force was almost guaranteed to fail. Denial and control of these places were important, and destroying and occupying castra was a major aspect of the crusade. In cases where fleeing to another castrum or town wasn't possible, we do observe a phenomenon of displaced nobles, survivors and mercenaries banding together to fight and survive in the countryside.

Sources

Primary

Peter les Vaux de Cernay, The History of the Albigensian Crusade, trans. by Sibly & Sibly

William of Puylaurens, The Chronicle of William of Puylaurens, trans. by Sibly & Sibly

William of Tudela & The Anonymous, Song of the Cathar Wars, trans. by Janet Shirley

Secondary

Elaine Graham-Leigh, The Southern French Nobility and the Albigensian Crusade

Laurence Marvin, The Occitan War: A Military and Political History of the Albigensian Crusade, 1209-1218