But it seems like this is ultimately a bad idea. What if the enemy runs through and destroys/takes everything left behind?

You've actually hit upon a major tactic. The English in the 100 Years' War commonly used a strategy known as chevauchée, moving quickly, trying to avoid attacking fortifications and pillaging everything they could on the way. As well as, hopefully, stealing lots of valuable stuff, this strategy aims to destabilise the enemy's position - if they can't protect an area, it hits their prestige and power, local nobles are going to start cutting deals or changing sides...

So what did castles do?

There are two key points here.

The first one is this. Today, people in rich countries have lots of possessions. There's a huge amount of infrastructure, factories, offices, vehicles, and so-on. Destroy this and you can cripple a country. In the Middle Ages though? The greatest wealth of a country was in its land. You can't steal the land, and it's pretty hard to destroy it. So even if an enemy does ravage your countryside, it's not necessarily a disaster; you can recover from it.

This is especially important in areas where raiding was common. These raids might be very quick, just there to grab whatever or whoever valuable they can then retreat. A castle gives you somewhere safe where you can keep yourself, your family and your key valuables. A good example would be the Marches in Britain, around the Scottish and Welsh borders, where there was a proliferation of small castles and fortified manor houses.

Second, why don't you just encircle the castle, outside of arrow range and sit around for as long as you want? Well let's consider who's going to be more comfortable. The people living in a nice solid building, with lots of stocks of food laid on. Or the people in an unsanitary camp in the countryside, trying to live off the land? Sieges were often nastier for the besiegers, facing weather, hunger and disease in their camps. Sieges could easily be harder on the attackers than the defenders. It's also going to take more men to besiege a castle than to defend it.

While your men are sitting there, your opponent can be marshalling their forces, manoeuvring against you, harrying your foragers, taking the initiative... Or maybe just waiting for your army to be crippled by dysentery, run out of supplies or simply run out of time and money (don't forget that all those besieging soldiers usually needed to be paid...).

So why can't you just ignore a castle completely? The garrison may be small, but even a small number of men behind you can be dangerous, threatening your lines of communication and forcing you to watch your rear. It was also not uncommon for an army to retreat upon a castle, meaning they might hold a more significant force than just a regular garrison. Finally, if a castle is on the border of your territory, as many castles would be, the garrison might also sally out and start raiding your lands.

It's also worth noting that castles weren't just important in wartime. They also had important peacetime functions as statements of power and status, administrative centres and more.

(Most information here from Warfare in the Middle Ages by Philippe Contamine, Medieval Warfare: A History, edited by Maurice Keen, and various threads here, like this one.)