Bypassing an enemy strong point during an invasion can put forces there in the position to cut your supply and communication lines. Or you have to deal with the possibility that forces from that position will attack your rear as you are attacking another position. But that would depend on whether or not there are sufficient forces in the position to launch these types of operations. Whether bypassing a castle would be a sound decision would have to be considered on a case-by-case basis. There is also the issue of what makes sense tactically versus strategically.