Finding a way to break the Phalanx. If a hole appeared in it (say the soldiers had to go round a boulder) and an opposing army could find its way through then the tightly packed soldiers were near helpless. The best way to do this was to force the Phalanx off the even ground that it required onto hilly terrain, eg. Battle of Cynoscephalae.

Flanking it. The Phalanx was incapable of reforming to face a new front quickly. If you could get cavalry or a flank attack going your odds of victory increased significantly, the Battle of Cynoscephalae is also an example of this.

Examples are more easy to come by from the Roman period as in the Hellenistic period and before almost everyone used the Phalanx. The Romans used the Phalanx until the Saminite wars when they switched to the Maniple (groups of about 100 men arranged in a checkerboard fashion as opposed to one long line). The maniple was much more manoeuvrable and did not break up on hilly territory but still kept many of the advantages of the Phalanx.