There are complicated reasons.

For starters, not all forested land is suitable for agriculture, even when cleared, soils in... coniferous? (sorry, I'm really not up to scratch on english botany terms) Forests like pine or fir can get very acidic, and often has low nutrient contents. There are other issues as well (soils being rocky, unwatered, swampy, etc.) that puts a certain limit on what can reasonably be used for agriculture.

Another important limit is fertilizer: Without modern synthethic fertilizers soil needed regular fertilization, mostly by manure from animals (or people, though in some cases seaweed was used, or fish carcasses, and some areas were lucky enough that the deposition of sediment from rivers meant that fertilizer wasn't as neccessary) or it would lose nutrients and yields would decrease. This lead to a certain tension between pastureland (which tended to have much lower yields than grain-growing, but was neccessary for the production of fertilizer that kept said grain-growing yields high) and cereal crops. As population increased what would happen would often be an increase in cereal cultivation (which would provide more food) but at the cost of long-term leeching of the soil, as there wasn't enough fertilizier, and thus long-term decrease in yields.

So a lot of increase in agricultural lands would come at the expense of pastureland, which would in turn slowly decrease the efficiency of the agricultural lands. (note that there were various techniques to mitigate this, at various periods, but it remained a fundamental tension for a long time) Note also that relying on a single, high-yield croup means you are very vulnerable in case of say, bad climate, or blight.

As for burning specifically, there is a technique called swidden agriculture, which involves burning down forests (the ash is fairly nutritious and thus acts as "starter layer" of fertilizer) however, without continous fertilization that will get used up after a couple of years.

That said, to a pretty large extent people were exploiting mariginal lands, new farms were set up, etc. But given the technology level and the needs there were simply limits on how far they could go.