Thirty men is actually typical of castle garrisons in the British Isles at that time. Edward I's Welsh castles, which include some of the largest and most advanced castles made, typically had a garrison of about 30 men, evenly split between men-at-arms and crossbowmen.

The very nature and design of a castle is to give every advantage to the defender. Avenues of approach are limited. In Stirling's case, the castle was (I say was because the castle that is there now is so rebuilt as to be essentially a different castle to what was there in 1304) built on a cliff. The crossbowmen could shoot from near-complete cover. The defenders could move from one part of the castle to another completely unseen by the besiegers. And a small garrison means your food supplies stretch farther.

Let's take your question of why the English didn't just scale the walls.

First, the besieging army wasn't gargantuan, maybe a few thousand. A large invasion force, like the English army at Stirling Bridge or the particularly large army at Bannockburn topped out at a high-end estimate of 25,000. We can assume King Ed had somewhere south of that number at the siege of Stirling. Let's say around 5-10,000.

That's still a lot of men--outnumbering the defenders maybe 300:1, but they're going to be restricted along one front. To get to the defenders, they'd have to:

Build a whole bunch of ladders. And not just any ladders, but ladders long enough to reach the top of the walls and strong enough to support the weight of all the men you're sending up.

Now they have to approach the castle walls. But first they'd have to contend with berms, ditches, and dry moats which will slow them down and make it more difficult to carry those ladders across. They could stop and level out that land to make getting the ladders through easier, but that will take more time. And as soon as they're in range, they're going to come under attack from the crossbowmen; crossbowmen they can't even see because they're hidden behind merlons or inside towers.

Still under crossbow attack, they now have to reach the walls and raise the ladders. Even the base of the wall is not safe, as the towers project so the crossbowmen can shoot along the length of the wall, the guys on top can drop stuff on their heads. There very well could have been hoardings--temporary wooden structures--built along the tops of the walls encasing the battlements. The defenders could step out over the walls and still be fully concealed and have direct line-of-sight to the base of the walls immediately below them. These hoardings also mean there's a solid wooden wall and roof with narrow gaps too small to get a man through but through which the defenders can shoot projecting a few feet in front of the top of the wall.

If they can get past the hoarding, or if there is no hoarding, they still have a small number of ladders along a narrow front of wall. If the ladders are just long enough to reach the top, they can be easily pushed over by the defenders, but the longer they make them, the more fragile and easily broken they are, and the longer they take to climb while under attack from above and the sides. The attackers could send an engineer up to drive an iron ring into a putlog hole and lash the ladder to the wall with rope, but that takes time while the men are sitting vulnerable at the base of the wall.

While climbing the ladders, the men are fully exposed, and thus vulnerable. If multiple men try to climb at once, that's a lot of weight on a wooden ladder. If one of them falls due to accident or injury, they could knock off men below them. Once a man reaches the top, he is now one man hanging on to a ladder on the outside of a high wall trying to push his way past likely multiple defenders with their feet on solid rock, long weapons, and murderous intent. They don't even have to kill him outright, just give him a hard enough shove. The attacking soldier is at a massive disadvantage until he can get his feet solidly on the wallwalk, and then he's outnumbered until enough of his friends climbing up the ladder behind him can do the same.

People are not mindless automatons, and have rather strong self-preservation instincts. Picture yourself standing below the walls with the attackers. The ladders are up, but no one can push past the defenders. One ladder breaks under the weight. Another gets pushed over. Crossbow bolts are thudding into ground and bodies all around you. Rocks are falling from above. Your friends are getting hurt and dying all around you. If you climb the ladder, you know there's a pretty good chance you'll get hurt and possibly die. If you stay put, there's a pretty good chance a crossbow bolt or rock or hot sand or boiling water will find you and you'll get hurt and possibly die. Multiply this by all the other hundreds of men around you, and retreat seems inevitable. The commanders could order a second attack, but that'd mean having to build more ladders, and the now-fewer men know what to expect and how hard and dangerous it is and would just be more inclined to retreat.

Taking it slow at a siege helped mitigate the advantage of the defender. If the besiegers can dig or use wooden barricades and roofs of their own to protect against projectiles, they can slowly reach the base of the walls safely. If they can force a breach in the wall, they can meet the enemy on a more equal footing (though the defenders still can plug gaps with barricades and force local superiority of numbers). Catapults can help do this from afar, but even that takes time.

Surround-and-starve combined with long-range bombardment looks more and more appealing.