If castles are force multipliers for the defending army why bother risking personal and resources to take it? Would it not be more advantageous to move on and face the castle’s forces in the open as your forces march and take other objectives? Or leave behind a smaller force to keep the castle under siege to starve them out while your main force moves on? Sieges can be costly to the attackers so it seems like they should be avoided whenever possible and whatever force the castle garrisons is better drawn into open combat or left to starve behind their walls.
It’s very hard to maintain a siege with a small force, you were often besieging a fortified town or city, not a just the fortress withIn it. A small force would be totally incapable of doing this.
Expecting a probably smaller and inferior force will come out and meet you in open combat on a field of your choosing isn’t going to end the way you like either.
Just leaving it and moving on isn’t likely to be much of an option either, as they can come out and attack your rear, flee, or send for reinforcements.
One aspect often not considered is that often the army inside is in a better position than the one outside in terms of resources. Unless you’re lucky and crops have just ripened (and not been burnt in the fields as you approached), you’re probably short of food and other supplies as that’s all been taken behind the walls. Attacking a fortified strongpoint was often as much about getting the resources held within as any purely military objective.