There are so many answers to this question. While /u/tim_mcdaniel laid out a bunch of important ones, there are still more. So, building on his list...

(8) When you've invested a castle (or city or whatever), you're scattering your forces around the location rather than concentrating them. This means that the defender, if sufficiently audacious, can stage small-scale raids ("sorties") to harass your line. And if they're successful, they can burn your stuff. Nobody wants to build a trebuchet only to have some defenders come out in the dead of night and burn it down.

(9) As an extension of (8), consider the disposition of forces. If I have 2000 men and you have 1000, I'm almost certain to win a battle in a field. But in a siege situation I now have to use my 2000 men to surround your 1000 men. This means that every single point in my line will be weaker than the forces you could conceivably muster to attack it. If my line is divided into ten points of roughly equal strength, then each of those links in my earthworks would be defended by only 200 men. You're able to take your strategically inferior numbers and concentrate them in a location, granting you a tactical (or localized) numerical superiority. This is basically the Schwerpunkt doctrine that Germany would use so effectively in WWII.

(10) There's no guarantee that the siege will be short. While the vast majority of sieges were short affairs or otherwise involved only a small number of forces (Ottoman governors in Hungary and Mesopotamia were notorious for conducting isolated sieges of border forts), there were a bunch of sieges that were lengthy, drawn-out affairs. The most famous of these lengthy sieges is Candia, a Venetian-held city on Crete that resisted a low-intensity Ottoman siege for 21 years (it eventually got to the point where the Ottomans were casting cannons in the field so they could fire Venetian roundshot back at the city).

(11) Logistics often made it extremely difficult to conduct a siege. A great example of this, albeit a pretty unique one, is the Ottoman siege of Malta in 1565. Malta is, from a military perspective, a pretty barren place; it has relatively limited running water (especially considering the army sizes involved), precious little in the way of forage, and obviously the armies there can't source gunpowder in the field. Everything either side needed had to be shipping in. The Ottomans were fortunate in that they had naval superiority here, and could thus ferry in supplies as needed, but that naval superiority didn't prevent the Spanish-led relief force from sneaking onto the island and then relieving the siege.

(12) Even if you win a siege, the cost of doing so can be utterly ludicrous. The same Ottoman commander that lost at Malta, Mustafa Pasha, would go on to win at Famagusta in Cyprus against the Venetians in 1571. That particular siege cost so much Ottoman blood that Mustafa contrived an excuse to disregard the surrender agreement and massacre the surviving Venetians.

(13) Capturing a castle often meant battering down its defenses. If the enemy then rolled along to attack the castle before the defenses had been repaired, you've effectively done their job for them. After all, if you blow up the other guy's wall and storm the breach, that breach is still there until you repair it. And if an enemy army is only a few weeks away and you want or need to hold the location, you've now got a wall with a huge hole in it. Makes their job easier, as you can imagine.

(14) I suppose there's also the question of glory. There's not much glory to be had in a siege -- if a traditional set piece battle is a gentlemanly game (it's not but that's how the romantic depiction of battles often go), then a siege is the equivalent of a back-alley brawl. And if it's a city, there's going to be more than one literal back-alley brawl before the fighting is done.

(15) As an extension of the above, trying to take a castle (or even just a fortified city) by storm was often a nightmarish prospect. Consider the kinds of things you'd be up against. The enemy would have murder holes on ground level so they could poke you with spears while you couldn't do anything to hurt them. They'd have hoardings and machiolations so they could dump stones and oil on you from above. They'd have embrasures in the wall that functioned like murder holes for their archers (or what-have-you). They'd have posterns hidden about so defenders could sneak out and flank you. And if you did manage to slog your way through all these hellish little traps and breach the gate, odds were that the enemy had more than one gate to deal with (and often the outer walls, which you've just spent so much blood taking, were shorter than the inner walls and thus exposed to arrow fire). In fact, attacking in siege warfare was often so ridiculously dangerous that volunteer units were often raised and promised huge rewards if they were successful (they were usually called some variation of "Forlorn Hope").

(16) Capturing a city, and to a lesser extent a castle, meant that your troops were effectively entitled to plunder and loot the city. That's usually regarded as a pretty ancient tradition. You see this crop up in 1453, for instance, at the Siege of Constantinople. Mehmet II repeatedly tried to talk Constantine XI into surrendering so that the city (which he had planned to make his capital) could be spared the looting. But he had to take the city by storm and even though he desperately wanted to maintain as much of the city as possible (going so far as to post troops at important sites, though not always before the looters got there), he was still forced to grant his victorious troops the traditional three days of plunder. I suspect that most commanders, even if they didn't intend to keep whatever they took, didn't look forward to effectively losing control of their men like that.

(0) Wasting resources if you don't succeed.

(1) Using resources if you succeed. It was not "wasting" if the castle was worth it.

(2) Opportunity cost. While your forces and resources are besieging Rouen, they cannot simultaneously be off besieging Le Mans.

(3) Warfare in general was much more expensive than peacetime. (Well, it still is.) If you're in earlier times and depending on your vassals, their cost-free service might well be limited to 40 days or so, so you either have to negotiate with them to remain (they might not want to, or they might demand money), or you have to hire mercenaries. Later, you tend to no longer have the free fighting men and are paying mercs from day 1.

(4) Illness. Until modern medicine, disease tended to kill more soldiers than combat killed, and cities were usually population sinks (more died than were born; they kept their numbers by immigration). A siege was a town or city without the established infrastructure of a town or city, limited though it might be. The soldiers may have improvised latrines draining into the water supply, befouled water, flimsy tents or shacks for protection from the cold and rain and not enough firewood, ...

(5) Supply in general. If you have a substantial force, you need to give them water, food, and other material -- a temporary town / city. But carting is slow and expensive, river travel is great for bulk but may not be available, the countryside might have been stripped of supplies by you or by the enemy, the foe might be blocking supply lines, in any event you're likely plopping more people in a place than it has hitherto supported.

(6) You might not manage a full siege. Plenty of times, the defenders managed to occasionally get supplies in or people out. Pretty useless siege then.

(7) If you're pinned down besieging a place, the enemy might be able to gather substantially more forces and besiege you from outside, and now you're stuck, cut off from supplies and sandwiched between two foes. I don't recall cases of this happening; I presume that if it was threatened, the original attacker fled or met the foe in the field.

In the Art of War, it is recommended that a general should never besiege a city unless it is the final option. The reasons for it are indeed mostly resources, but also the time. It takes much planning and organization for a siege to go well, and without planning the besieger are almost sure not to win. Besides that, it would be easy to be attacked by surprise, demoralization would run rampant through the besiegers camps as it takes a long time (no one likes doing anything for months on end), and disease would likely run rampant through the camps. Overall, it just tactically seems like an inferior option. There may be other historical reasons, but they may all stem from the reasons listed before.