In ancient battles, on the whole, the heavy infantry of the main battle line was very much focused on the fight in front of them. The anticipation of having to engage in close combat was mortifying, and the business at hand required their complete attention. In addition, given the minimal technological means of conveying orders, ancient generals rarely issued new orders after the battle had begun, relying on their men to carry out the task that their deployment set before them.

As a result, when an enemy force suddenly appeared in their flank or rear, they got confused easily, since they no longer knew what was going on; morale would collapse quickly, and men would scramble to take advantage of whichever way out was left to them. If the encirclement was complete, as at Cannae, panic and chaos would soon destroy any remaining capacity of the encircled army to resist.

Good discipline and effective local leadership were decent remedies; veteran Roman armies like that of Caesar were generally able to respond well to being outflanked. Greek hoplites, on the other hand, were much less well-trained than Roman legionaries, and their behaviour in battle was correspondingly fickle. There are many examples of entire armies routing when an enemy force appeared in the flank or rear of the phalanx. It needn't even be a large force. At the battle of Delion, the Thebans sent 200 cavalry to save their crumbling left wing; the Athenians, thinking a whole fresh army had come up to reinforce their enemies, promptly fled.

Besides the points u/Iphikrates mentions, it also appears that soldiers in ancient battles needed a certain amount of space to be able to fight effectively. If they were clumped together too closely, they would be unable to use their swords, shields and javelins effectively and would find themselves unable to mount an effective defence.

Caesar proceeded, after encouraging the tenth legion, to the right wing; where he perceived that his men were hard pressed, and that in consequence of the standards of the twelfth legion being collected together in one place, the crowded soldiers were a hinderance to themselves in the fight; that all the centurions of the fourth cohort were slain, and the standard- bearer killed, the standard itself lost, almost all the centurions of the other cohorts either wounded or slain — Caesar, Commentarii de Bello Gallico, book 2, chapter 25

This is exactly what happened at Cannae. The Roman force was very numerous but not very well trained, on account of it having been only recently recruited, and was deployed in a relatively narrow but very deep formation. When attacked from an unexpected direction, men's natural impulse is to clump together, to seek safety by the presence of allies, and to shy away from the attack. When attacked from all sides, this led to them being pressed together so closely that they were unable to defend themselves.

As long as the Romans could keep an unbroken front, to turn first in one direction and then in another to meet the assaults of the enemy, they held out; but the outer files of the circle continually falling, and the circle becoming more and more contracted, they at last were all killed on the field. — Polybius, Histories, book 3

Still, just because flank attacks could sometimes have a devastating effect, that doesn't mean they always did. Philip Sabin argues that their significance is sometimes exaggerated, citing the Roman victory at Ibera where they easily dealt with being outflanked, or the battles of Telemon and Ruspina, where armies fought well despite being attacked from two sides at once. This ties in to Iphikrates' point about discipline and local leadership being the key to dealing with flank attacks.