This was due to the added cost and labor involved in building fortified walls to enclose a large space, and the difficulties of defending a larger area. Moreover, castles were usually built into landforms that were difficult to assail. Large arable fields are usually found on plains that are easily approachable.

Economist here. In video game farms are show as this miniscupe plot of land, so you might get the idea that they can fit into a castle. In reality the land needed to maintain food supply untill the next harvest vould be quite vast. Even if you theoretically build such a perimiter it would be quite expencive to maintain and near impossible to defend. Also castle need solid foundations, whereas farmland needs fertile grounds. The biggest problem with this, however would be the feudal system and the taxation being paid in kind. Even if you are a powerfull barron, you'd still be going against established laws and way of life.

Simply put, it's easier to abandon the farmland and defend the ready product. You could have more that five years' worth of supply stored, which was more or less reasonable.