Briefly: because they weren't allowed on better land.

Very very roughly pre-industrial people can be divided into three groups: non-food producers (political leaders, craftsmen etc, comprising 1-10% of the population), farmers, and pastoralists. Non-food producers aren't relevant here.

In pre-industrial societies farmers produced the most food per unit work. However, there was a VERY limited amount of land that could be effectively cultivated before the advent of fertilizer and mechanically pumped wells so farming tended to be confined to fertile river bank, valleys, etc. The remainder of the usable land was typically given over to pastoralists: migratory or semi-migratory people who raised animals (sheep-herds are the typical European example) or just left wild. Often, but not always, these people were part of socially marginalized ethnic groups and considered somewhat lower class. There was (and is, in some places) lots of conflict between pastoralists and farmers.

But fundamentally farming was more productive, leading to higher birth rates. So where was a 5th child of a farmer to go? Their family land (which they probably did not own but worked on behalf of a landlord as a sharecropper) would likely not go to them. Instead, they had to seek out less productive farmland on the borders of the town or become pastoralists. Since pastoralism was its own type of skilled work (for which the farmers child would have little training) and because of social pressure most would likely choose to be farmers on worse land.

In general settlers on this worse land did struggle more than their better off peers but not everything was bad. While land far from a river is likely to be less productive than river farmland it is also less variable due to river floods. Over time farming technique could be adapted to a less water and nutrient rich environment producing yields that were only slightly worse per unit time invested.

With the advent of even limited mechanization (particularly powered irrigation) this gap became even smaller as farms could exploit groundwater to make up for their lack of surface water resources. Nation state policies in countries like the 19th century US further encouraged broad settlement in otherwise arid zones.

In all cases the arid-environ farmer probably does not want to be there. They likely view their societal position as marginal, just above the migratory pastoralists on their rural flank but well below the (relatively) wealthy farmers in established productive regions. They may face strong societal pressure to maintain an agricultural lifestyle and lack the know-how to pursue a different way of life.