Castles don't just overlook cities, if an enemy doesn't want to waste time laying siege (which, to the contrary of what Hollywood would have you think, consists sitting outside and waiting for the inhabitants to starve to death unless you're very very stupid) castles can offer valuable vantage points from which to harass enemy supply lines, as well as overlooking important roads, bridges, or choke-points. The decisive battle between the Lombard Cities and the Holy Roman Emperor Frederick Barbarossa, for example, took place at the foot of the Castle of Legnano, a fortress on a vantage-point overlooking a ford over the Olona river, which allowed the Milanese to remain supplied through the battle.

Many urban castles developed out of a settlement's oldest and most defendable neighborhoods; for much of its early history, the city of Prague was entirely contained within Prague Castle. In later centuries, the castle's position on the hills overlooking the "Old Town" and "New Town" made the city difficult to isolate. In Paris, the Louvre Castle (before its extensive restructuring) overlooked the oldest bridge on the Seine, cutting the city in half. Many large cities didn't even have castles, and often when they did they were more to keep the ruler safe from angry mobs than anything; the castle of Milan, although massive, never saw a protracted siege, while Pavia to the west was the site of multiple major battles. Similarly in Rome, it was the habit of Popes to lock themselves in the Castle of St. Angelo in times of crisis.

Bypassing a castle left your supply lines at risk.

It could also leave your troops and command centers in a perilous situation as typically castles are amazing bases for raiding from giving the defending army multiple advantages to attack your armies weak points such as the flank or rear as you try to avoid the castle.