The strength of a stone wall is in its weight. Stone has enormous strength in compression, but little tensile strength, and mortar even less. If you were to make a chain out of stone and hang something heavy from it, it would not be long until there was a crack forming and the chain would break ( yes, geologists, there are exceptions...but there is not that much jade in the world). So, in a castle wall the immense weight of the stones above held the stones below in place.
Medieval people were not stupid, and this weakness of a stone wall was known. There were several techniques for attacking a stone building. A corner was a weak point: a party of workmen could attack a corner with hammers and picks to create a notch. Once the notch got big enough, the stones above could break free and fall, and possibly a good bit of the walls on either side. That's why earlier castles are square, but later ones are round ( and also why you'll still see, in old cities, stone, cast iron or concrete guards on the corners of brick and stone buildings to keep turning vehicles from hitting the masonry and starting a dangerous notch).
Another method would be to mine under the wall, propping up the mine with wooden bracing. Setting fire to the bracing would let the mine collapse and the wall above would follow. This was another reason why a moat was useful: not only is it rather hard to set ladders into a moat if you want to get armed men over the walls, it is very hard to dig a mine underwater.
The reason to start a fire outside would be in order to attack the door. Doors and gates were major weak points, which is why you see rather complex structures of double portcullises, drawbridges, and yetts to control entry, and hoardings and machicolations to defend them from above.
And, by the way, once the castle was taken, it was not that hard to "slight" it, render it indefensible. A party of workmen could simply start taking apart a wall from the top, stone by stone, where there was no weight from above to hold them all together. And of course it's much easier to push stones off the top, than it is to carry them up.
Yes, undermining the walls or towers was one of the major ways to breach a castle wall in medieaval (and earlier) siege operations both in the West and the East.
The most "primitive" way is to just go at the base of the wall with picks and other implements, preferably with somekind of protection like a movable roof or mantlet. Though at Acre in 1191 the sappers worked unprotected, which would be incredibly dangerous. The wheeled battering ram or bore with a covered roof is an example of taking this general idea and running with it.
While removing stones at the base of the wall you want to shore it up with timber and then fill the cavity with combustibles and set it on fire taking the wall down with it. Unsurprisingly the defenders will do their very best not to let you work at the base of the wall. So the base is usually thickened or otherwise strengthened to resist collapses. The other thing defender's can do is to dig a moat which makes it harder for attackers to get to the base of the wall with machineries (including siege towers). If practicable you want a wet moat since this makes mining much more dangerous as the moat is liable to drain into the mine, which not only is lethal to the miners but also lets you know something is happening.
So working at the base of the wall is quite feasible, ideally you don't want to advertise the point that you are attacking because it lets the defenders know where to concentrate their efforts and can even let them build secondary walls and palisades. You accomplish surprise by creating a mine and sap. This basically means you dig down deeper and probably further away, ideally from a spot that cannot be observed. Inside a house nearby is e.g. ideal and one reason you don't want buildings close to your castle. Reaching under the foundation of the walls you dig out a gallery which is supported by timbers and again you fill this with combustible materials to burn off the supports and collapse the above structure into the space you dug. I should note it's not the incendiaries themselves that are doing the job, they are there to quickly collapse all the supports without yourself having to be anywhere near the collapsing mine and walls. Now the arrival of gunpowder on the scene changes this of course, the explosion being the point of the mine and lets you do quite spectacular things if you got the time and gunpowder to burn.
Ideally you want your castle built on bedrock to avoid mining. At least in a feasible timeframe.
If you didn't have bedrock underneath you then you tried having the wet moat, but defenders also used counter-mining, i.e. digging your own mine under the enemy mine (especially when gunpowder because part of the equation) or into it usually resulting in vicious tunnel fights. Or as at Dyrrachium in 1108, where the French mine was frustrated by the Greeks breaking in and piping Greek Fire into the enemy miners. Sometimes countermines were dug in advance too so the defenders could relatively quickly hit any enemy mines. A long siege could be characterised by a multitude of mining and counter-mining operations. The counter-mining can of course be detrimental to your defence, but ultimately you have to stop the enemy form successfully completing amine of their own.
In the 14th century Lord Burghersh besieged the castle of Cormicy and managed to mine it. He then invited the knight in charge of the castle's defence to come view it, which prompted the well-supplied defenders to surrender without the mine having to be fired, the end result not being in question at that point. That's how dangerous a mine could be to a castle.
I'm using Christopher Gravett's "Medieaval Siege Warfare" (1990) where I picked the examples from.