At the end of the day there was an army inside of the castle. They didn't just stay in there, they went out and attacked the army in the field. Moreover, they get a pick of battles. Don't like the look of the fight today? Go back inside the nice, safe castle and wait. Did you catch someone out of position today? Good, cause some casualties and get back inside before their buddies can show up. Besides, every day those guys are marching around or waiting in a field chasing after a fight that isn't likely to come is a day they aren't farming at home.

Moreover, the castle was an effective base. It had months or years of supplies, it had long standing hygienic infrastructure, and generally had whatever craftsperson needed to fix up what breaks. An army camped out in the field was often at a significant disadvantage. It had to forage for local food or haul food in from elsewhere, this was an expensive and time consuming problem. Disease is also a common problem of armies that stop for any appreciable length of time in a single spot, it wasn't terribly uncommon from strong armies to be critically weakened by an outbreak of some plague or other.

In short, having an army in arms out in the fields was expensive and dangerous. Having an army in a castle was less so. At some point the attackers would run out of money to keep men in the fields or run out of men in the fields and would have to leave. It was faster and more efficient use of everyone's time to just hit the castle.

Also, the occupation of land wasn't always the goal. We aren't talking about territorial nation states here. It wasn't uncommon for multiple people to have authority over the same land, so having troops on the land didn't necessarily mean exclusive control over the people, wealth, honor, or political authority that the war was actually being fought over.

You may find this thread in the other current castles discussion enlightening.