Would the church, or the feudal Lord, decide to start one? Where would they get the nuns or monks from - would people transfer from existing ones, or would everyone be new?
I'm fairly familiar with the history of some 20th century religious communities and how they were founded, so I'm interested in how that compares to the administrative legwork for founding them historically. (I was mostly thinking of the Middle Ages, but to be honest I'm interested in any time period.)
The most common way of founding a monastery in the middle ages was this. When the patron, a ruler, secular lord or ecclesiastical lord, decided to found a house they would open negotiations with an existing monastery of the desired order to obtain staff to colonise the new monastery. In the case of centralised orders like the Cluniacs or the Cistercians, they would talk to the authorities of the order.
The patron would be expected to provide a suitable site, and erect there a chapel, accomodation buildings, farm building and gatehouse so that the new community could begin its life immediately upon arrival. These buildings, often of timber, would be temporary, to be used while the permanent monastery complex was under construction, a work that could take decades. The patron would also have to give an initial endowment of estates to support the community and grant a charter confirming them, and would usually give funding for starting work on the permanent buildings. All these details would be subject to lengthy negotiations.
In return, the mother house or order would provide qualified staff for the new house. The canonical minimum was 12 choir monks/nuns plus a boss, to reflect Jesus and the apostles and provide enough bodies to fill all the needed positions and perform services adequately. A force of laybrothers might also come along in orders (eg Cistercians) that had them. The new community would be recognised by its peers and the church authorities as a legit monastery.
There were other ways. A monastery might develop organically, often from a community of hermits or the followers of a religious leader who attracted donations from the pious. The clergy of an existing major church might decide to become monastic, they might also be forced to by the government or a local bishop (this happened a lot in the 10th-11th centuries during the 'Benedictine Reform' movement). A lord might donate a church that they controlled to an existing monastery on the basis that the monastery set up a monastic community there. New monasteries could even result from infighting in an existing monastery, usually when a faction was dissatisfied and voted with their feet, going off to found a new house more to their liking. Fountains Abbey in Yorkshire is the most famous example of this.
J Patrick Greene 'Medieval Monasteries' (Leicester University Press, 1992) has a great introductory discussion of the foundation process.