Keep in mind that the concept of direct action (in military science called 'decisive battle') is a relatively recent development, only begun in any meaningful sense in the mid-18th century and can be attributed to Frederick the Great of Prussia. Prior to this, armies would in fact seek to avoid confrontation. They'd move around the countryside, wrecking havoc and raiding for plunder and supplies.

In medieval warfare, it would depend on the objective. Since armies themselves would rarely sally forth and seek to decisively engage and destroy one another, taking castles would only be attempted if it served a wider purpose. Operational art was not developed yet, and wouldn't be a factor in warfare until the 19th century, but strategic goals have always been present. If the goal was to take territory, a castle would be besieged and assaulted if it was deemed as necessary to 1) dislodge a foe from the land or 2) prove a point by defeating an enemy force. Grand battles were a very rare occurrence, which is why wars could last decades in some instances; there was rarely a decisive outcome due to the fact that armies tried at all costs to not square off against each other.