What are the mechanics of the square formation (combined with sufficiently long spears of course, or sufficiently powerful and fast loading muskets) that makes the cavalry unable to penetrate it? Or is it simply the square versus the line concentrate enough combat power to stop the cavalry?
At its most basic level, horses aren't stupid. They will not charge home against a steady line of sharp pointy things -whether it's a mass of pikes (eg a Schiltrom in Scotland) or an infantry square. If there are gaps or its loose, all bets are off. To break a square or Schiltrom with cavalry it helps to break up the cohesion of the unit so arrows or cannon or musket fire will do the job and the cavalry will then be able to do the rest.
To take a couple of examples:
Falkirk 1298 - The Scottish army was divided into four schiltroms of massed spears to protect themselves against the English heavy cavalry. This worked for some time with the English cavalry unable to penetrate or damage the schiltroms (the unprotected Scottish archers were ridden down). Welsh archers were then moved up close to the schiltroms and began firing. Once the schiltroms lost cohesion the cavalry were able to get into the schiltroms and break them up from the inside out.
Garica Hernandez 1812 - Dragoons of the Kings German Legion charged a French square of the 6th Legere. The square held off firing and when they did fire a dead horse and rider fell into the square collapsing the ranks. The subsequent riders took advantage of this and rode into the square. The square broke and many of the men ran towards the closest square followed by the KGL dragoons, the panic resulted in gaps opening in the face of the square as men fought to get in and the KGL dragoons again broke the square from the inside out.
A square in situ (disciplined and trained troops could march while in square - see the Light Division's retreat at Fuentes de OƱoro in 1811) would be able to present four faces to the cavalry. Multiple squares were positioned so that they were mutually supporting (think of a chess board). Cavalry riding at a square would meet a volley at some 100 feet away and then a rolling volley would run along the face(s). The resulting piles of dead and wounded horses and men would obstruct the following ranks of cavalry and subsequent charges would bog down. Unsupported cavalry could ride around for hours (and they did at Waterloo) causing little damage beyond pistol shot. For cavalry facing disciplined troops in square they would ideally be supported by artillery (especially horse artillery) which could unlimber within range of the squares and concentrate fire to create holes for the cavalry to exploit.
For sources on the square (and how to form it from different marching and fighting positions) see:
A New System of Military Discipline for a Battalion of Foot on Action
National Militia Standard, Embracing the Discipline of Infantry, Light Infantry, Riflemen, Light Artillery, Horse Artillery, Cavalry