It depends on what you mean by 'phalanx'.

If you mean the Macedonian/Hellenistic pike phalanx - a tight, deep formation with overlapping ranks of pikes - there are several reasons why it went out of style.

Firstly, the pike phalanx was really only suited for pitched battles on open ground in which both sides fielded a battle line of heavy infantry. It needed open space to function properly. In any other combat situation - in broken ground, during sieges, or against more mobile troops - large formations of pike-armed infantry could not be deployed to good effect.

Secondly, effective pike combat was the prerogative of well-drilled professional soldiers. Only states with the financial means to support large standing armies could field significant numbers of good pikemen, and of course a pike formation that was too small could serve no useful purpose.

Thirdly, for such an expensive weapon, the pike phalanx was simply too vulnerable. Aristotle already complained that a tight phalanx could be torn apart by the slightest irregularity in the terrain, making all of its members vulnerable to infiltration and attack from up close, where their pikes could not reach. At the battle of Issos, a last-minute oblique manoeuvre exposed the sides of Alexander's pikemen to the enemy's Greek mercenaries, who gleefully began carving them to pieces. Several major Roman victories against Macedonian pike formations were due to the legionaries' superior tactical flexibilty at the small unit level. At Pydna, they simply lured the pikes onto a rocky hillside and butchered them from up close as the formation crumbled. Ultimately, a more flexible, multi-purpose soldier like the legionary was a better investment.

If, on the other hand, you mean the Greek phalanx - essentially just a somewhat organised formation of well-protected spearmen - then the answer is that it never really fell out of use. Spearmen with large shields can be effective against both infantry and cavalry even with minimal training. The Gauls and the Germanic tribes faced by Rome often fought in this manner, and the Romans themselves adopted it later in their history, especially against cavalry-heavy armies in the east. Later formations such as the Viking and Saxon shield wall work by the same principle.

/u/Iphikrates has already noted some of the notable problems with the Phalanx. I'll only add that the original Greek Phalanx was never that successful to begin with when used on its own - It was always just one component of a much larger military machine.

It pays to remember that the Greco-Persian Wars and the Peloponnesian Wars were in many respect naval wars involving numerous naval battles and expeditions. Thermopylae for instance was fought at the same time as the almost-forgotten naval battle at Artemisium; while the most decisive campaign of the Peloponnesian War was Athen's disastrous expedition to Sicily.

Indeed the Greeks themselves were ultimately conquered by the Macedonians - whose army made decisive use of cavalry and other arms in all of its battles; in addition to the Phalanx.

However the traditional Greek phalanx really began to go out of the vogue with the rise of Rome. The Romans appear to have incorporated the Phalanx in early republican armies - and even their later incarnations included a pike block in the form of the last-resort veteran line of triarii. However, this all changed in the Imperial era - where the Roman Legion rapidly recognized that it needed to be highly adaptable to face the multitude of enemies it faced.

The Roman Legion was in fact not just a heavy infantry force with pilum and short sword as often depicted in video games, but would employ a variety of tactics (archery, pike formations, heavy infantry, etc) depending on the enemy supplemented by Roman engineering, siege artillery, and local auxilia.

In a sense, the Phalanx stopped being "viable" because it was now part of a larger and more diverse world; and sticking to this tactic wherever you went was suicide when you could be facing Parthian horse archers one year and Gallic infantry in another year. Phalanx-style formations and pike blocks would continue to be used - and indeed the Swiss would later forge their reputation by the use of these tactics - but it was never the be-all, end-all fighting formation and was always part of a larger whole.

A recent post on the subject.

Edit: An older one with more responses

You're only thinking of melee weapons. Ranged weapons, especially from mobile forces, such as light infantry or mounted archers in later periods made tight heavy infantry formations ineffective. The Romans learned this the hard way during the battle of Carrhae against the Parthians. The Parthian archer cavalry could retreat and volly repeatedly whittling down the roman infantry without much loss on their side, and it's one of the reasons the battle was such a lop sided defeat for the Romans. Phalanxes really weren't an effective formation in and of themselves, but situationally could be very useful against certain other formations and types of forces, especially against cavalry armed only with melee weapons.

The whole scenario I outlined above actually replays itself later in history again when in the 17th century the main infantry formation in most of western Europe is the spanish tercio formation with pikeman and spearman in ranks with crossbowmen and musketeers thrown in as well as pepper the enemy. At the time many military leaders thought that its advent meant an end to mounted combat and the end of cavalry as a mainstay fighting force, but during the 30 Years War Gustav Adolph introduces linear tactics of well trained musketeer regiments and they decimate the ranks of the tercio formations allowing for both infantry bayonet charges and cavalry flanking maneuvers.

And it plays out again in post Napoleonic combat in WWI where infantry organization and action needs to be reorganized on the squad level due to the proliferation of better ranged weapons (the machine gun).

Basically, tight slow moving infantry formations are not a good tactic except situationally, but especially not when you're outgunned.