From all that I've read, the sieging of cities can be over rather quickly, even if they are huge, important cities like Baghdad - the siege took 5 days before the garrison surrendered to Hulagu, despite Hulagu's force barely outnumbered the defenders two-to-one - whereas the sieging of castles is long and costly. What usually made the sieging of cities so easy for the attackers? I understand that for one, the defenders have to cover a great area of walls, as cities can get rather large, but shouldn't this be solved by just having a larger garrison size?
You seem to be looking at a very limited sample. The siege of Calais in 1346, for instance, took eleven months, while the siege of Acre during the Third Crusade took almost two years. There are countless other examples of towns and cities lasting for months or even years.
As to the siege of Baghdad, it seems that a large percentage of the defenders were killed or cut off during a sally prior to the siege proper, and the Mongols were subsequently able to storm and size a large section of the city wall within eight days of the siege beginning. Unable to retake that section of wall with the limited number of troops he had left, the caliph had no choice but to surrender.
Candia.
That ridiculous example aside, it varied. But an interesting case is Hormuz in the 16th century. As part of a broader clash between Portugal and the Ottomans, the Ottomans sent a fleet to Hormuz. It captured the city but lost siege materiel in the process and the Ottomans had to retreat; the commander of the expedition was executed. This ended the career of Piri Reis, a talented admiral who was also a famous cartographer. Why? Because the failure to capture the fortress meant the Ottoman campaign failed. Hormuz reverted to Portuguese control shortly thereafter.