The primary purpose of a castle is not to withstand a direct siege infinitely, it is to function as a fortified elite residence, and act as a base of operations from which mounted soldiery can exert political and military control across a wide area. In war, castles are rarely intended to fight alone; they are instead distractions and delays, meant to hold the enemy up and provide enough time for a relief force to be mustered and meet them in open battle, where heavy cavalry can be used to maximum effect. One of the advantages of castles is that they allow a small garrison to hold off far greater numbers; the bigger your castle gets, the harder it is to defend, the bigger the garrison it requires and thus the more supplies it needs. Bear in mind as well that your castle needs to house everyone, as well as kitchens, stables, armouries, wells, pantries, etc. If you've got enough space in your bailey to construct a trebuchet with room to operate it safely, you've got a big castle.

One of the most consistent and defining features of castles is that they tend to be static. This makes them quite easy to shoot at as they can't really get out of the way. This means that your engineers can sight your trebuchet once and that should suffice. Returning fire against a mobile opponent, on the other hand, requires the trebuchet to be re-sited - essentially partially deconstructed and rebuilt - every time your target moves. Even if a trebuchet was pre-sighted, you'd perhaps get a single shot in before the enemy figured out what was going on and just avoided that general area.