Frankly, an army that is completely surrounded and flanked on all sides is generally doomed. The battle in game of thrones was probably inspired by the Battle of Cannae, in which the Carthginians completely surrounded and annihilated a much larger Roman force. The Romans were completely wiped out, and the Roman Republic very nearly fell apart from the loss of their entire field army.

The only effective way to defeat this tactic of Encirclement is to prevent yourself from being surrounded in the first place. Encirclement is such a disadvantageous position to be in that the only possible move when trapped in such a situation is to attempt a breakout - organized tactics are impossible to carry out when being attacked from all sides in traditional formation combat.

Now, if the shield wall formation was -not- encircling the enemy army, the most effective tactic would be to simply flank them, or use skirmishers to cut down the formation from a distance - a direct charge into a 'phalanx' formation or other heavily reinforced spear or pike formation was near impossible to carry out successfully. The only way to do so would generally be seen in the tactics of Persian and Greek battles, in which a frontal attack on a phalanx would be carried out by -another- phalanx. This would quickly become a battle of physical exertion and endurance, probably ending when one side was sufficiently bloodied or had their formation so broken that their phalanx was rendered useless.

Not an answer, but a slight variant on the question: Can we validly compare that battle scene to some battles between Swiss phalanx and Spanish tercios? I could easily be wrong about the tercios, but I am under the impression that the Swiss mass-pokey-thing formations were defeated by folks using short chopping weapons, who would attack the wooden handles of the pokey-things until they could get inside the reach of the pokeys and then start working on the folks holding them.

(I am using silly, generic terms in an effort to not get bogged down in terminology.)

It is hard to provide a direct answer to this question, simply because shieldwall/schiltron/phalanx formations have been utilized in many different ways by many different armies over many hundreds of years. However, at their core, such formations draw strength from the unity and cohesion of the men within them; as such, countermeasures have to be primarily focused on degrading the ability of individual soldiers to effectively work together within these larger groups.

If you want a specific example, one of the more popular methods by which Renaissance armies sought to disrupt and destroy an enemy 'phalanx' was to equip a number of soldiers with close combat weapons, such as halberds, greatswords, or, in the case of the Spanish Rondeleros, swords and shields, and utilize them as 'shock troops'. These men would be tasked with the incredibly dangerous job of hacking their way through the bristling wall of pikes and engaging the otherwise ill-equipped enemy pikemen directly, in order to create chaos and confusion in their ranks and thus weakening their overall effectiveness as an organized unit; at the Battle of Ravenna in 1512, such tactics allowed the Spanish infantry to inflict massive damage on several columns of German landsknechts. Interestingly, this tactic was actually attempted by Tormund and the wildlings during the Battle of the Bastards, although with limited success; had they made a more concerted effort to break through the Bolton lines in this way, they may have avoided getting crushed together like they eventually did.

If you are interested, here is an image of the Battle of Kappel, in which a number of soldiers armed with Zweihänders are shown to be engaged in this very tactic.

Edit: Check out this scene from the Spanish film 'Alatriste'; discount the fact that this battle took place in the 17th Century and try to ignore the fact that it is obviously filmed on a rather limited budget, and skip to 2.25, when you see an interesting, if overly-dramatised example of how these 'shock troops' may have worked during an actual pike battle.

Follow-up question: Jon's initial plan was to feign a retreat in the center in hopes of over-extending the Bolton lines and allow a cavalry break though. Wasn't that a textbook tactic to defeat a phalanx?

As an additional question:

Before the "shield encirclement" maneuver, the battle had devolved into a chaotic free-for-all with no discernible lines and with soldiers on both sides being randomly all over the place and fighting one another almost individually.

It's a common trope in movies and is a convenient way to showcase the hero's fighting skills and bravery, but was it something that could realistically happen during ancient or medieval battles? It doesn't seem realistic to me.

To add something that hasn't been brought up yet. Heavy javelins could sometimes be enough to break up a shield wall. Julius Caesar describes this in an encounter with the Helvetii:

The Helvetii having followed with all their wagons, collected their baggage into one place: they themselves, after having repulsed our cavalry and formed a phalanx, advanced up to our front line in very close order.

Caesar, having removed out of sight first his own horse, then those of all, that he might make the danger of all equal, and do away with the hope of flight, after encouraging his men, joined battle. His soldiers hurling their javelins from the higher ground, easily broke the enemy's phalanx. That being dispersed, they made a charge on them with drawn swords.

This wasn't always enough though. The Romans were frequently struggling against Macedonian pike phalanxes head on, such as as the battle of Cynoscephalae.

Formations like that are slow to move and inflexible, if you can outmanoeuvre them then they lose their advantage.

You can also attack while they are moving into position or advancing as their formation will be disrupted, especially if moving across uneven ground. This was a problem phalanxes faced in battles because even a slight break in the formation will end very badly for the army.

The romans moved away from phalanxes early on because of this to less rigid formations that made them more adaptable. They never actually used pikes though, the spears they originally used before moving to swords were around 2m long and pikes are generally much longer.