I was rather hoping someone would have posted a period description, here.

I can say something about the basic materials involved. Masonry has enormous compressive strength, but comparatively little tensile strength. Especially the lime mortar of the time, which is softer than the Portland cement/mortar mix you find now. It is not too hard to pry a stone from another mortared stone. This was known: the reason castles eventually went from being square to being round is that it was possible for a beseiging army to attack a corner and get a notch started. If that notch went far, the enormous weight of all the stones above it ( which usually plays a big part in making a stone wall strong) would cause all the joints to fracture and the wall could collapse. Some castle walls had a fat buttress at the very bottom also, to try to prevent this: like at Alnwick castle.

Anyway, a crew standing on the top of a stone wall with hammers, wedges, prybars, and some time could get at the joints and separate stones easily enough. Add a hand-operated winch on the ground and some rope, perhaps whole sections could be pulled apart . Gravity would of course take them away once they were separated. If it was a smallish castle, something more like a fortified house, like Dunstanborough, it might not take them that long. And, of course, a castle is only defensible if it's got all the walls intact. If one side of it was destroyed, it was as good as useless. So, "tear down" might have really meant , damage enough to make it useless.