Macedonian phalanx vs. Greek phalanx

• Not accounting for cavalry and other forces would force me to speculate. The Macedonian phalanx in the early Hellenistic period was meant to work with cavalry. Reason being is phalangites were meant to hold the enemy lines in place while heavy horse circled in and smashed the rear or flanks. This is often seen in the famous "hammer and anvil" maneuver. Armed with long sarissas, the Macedonians were nigh invulnerable from the front, especially with a longer reach on the traditional Greek phalanx. If the two phalanx styles were to fight (and no units were to support them), it likely be phalangites that would win. Macedonians had longer spears, generally better training, and lighter armor (greater mobility).

TL;DR: bigger pokey sticks = good idea

Manipular formation vs. Macedonian phalanx

• The Macedonian phalanx at the time of Roman conquest were used much differently. Phalanxes now charged enemy lines instead of simply holding them. The manipular formation was meant to "bend" in a sense. The spacing in the formation would allow front line troops to fall back into rear lines. The pre-Marian field deployment would be front line hastati, who could fall back into more experienced principes, who could fall back into the most experienced, triarii. This meant the line could give in certain spots without breaking. The phalangtites, however, were very vulnerable to flanking and could be broken. During the Macedonian wars, the Romans did not fight the phalangites head on, but instead used the flexibility of the manipular formation to force then into rough terrain. The clumsy movement of the phalanx would cause their formation to break up, and they would lose the near-invulnerability that they had from the front. Roman troops would fill the gaps in the enemy formation and would wreak havoc. Once they were past the bristly front of the sarissas, Roman troops enjoyed superiority of arms.

TL;DR Maniples = tough pliable metal, Macedonian = hard, but brittle metal

Manipular formation vs. Greek phalanx *The traditional Greek hoplite was heavily armored. This is one of the disadvantages they had against they phalangites who wore lighter armor and had greater mobility. According to Polybius, the Romans wouldn't meet a cohesive phalanx head on. Instead they would try to disrupt the formation and than fill gaps. This was often done by pulling back and encouraging the phalanxes to follow. Since the phalanx was a clumsy and slow formation, they would often lose cohesion when moving rapidly.

Part of the problem you are going to encounter with this question is that the sources aren't really going to give you a good account of "who would win" style fights. It is fairly difficult to find an engagement where you can pin down exactly what kind of equipment is being used by both sides and a detailed description of how that engagement went.

For example, the battle of Chaeronea, in which Phillip of Macedon defeated an allied Greek army was one of the most important in the ancient world and probably the definitive phalangite vs hoplite battle. Yet, the sources for this battle not great at answering your question. The historian Polyainos, who wrote sort of bite sized strategy snippits, has two for the battle:

"Engaging the Athenians at Chaeronea, Philip made a sham retreat: when Stratocles, the Athenian general, ordered his men to push forwards, crying out, “We will pursue them to the heart of Macedon.” Philip coolly observed, “The Athenians know not how to conquer:” and ordered his phalanx to keep close and firm, and to retreat slowly, covering themselves with their shields from the attacks of the enemy. As soon as he had by the manoeuvre drawn them from their advantageous ground, and gained an eminence, he halted; and encouraging his troops to a vigorous attack, made such an impression on the enemy, as soon determined a brilliant victory in his favour."

and

"Philip, at Chaeronea, knowing the Athenians were hot [eager to fight] and inexperienced, and the Macedonians inured to fatigues and exercise, contrived to prolong the action: and reserving his principal attack to the latter end of the engagement, the enemy weak and exhausted were unable to sustain the charge."

Another source for the battle, Diodorus Siculus, has this to say:

"The armies deployed at dawn, and the king stationed his son Alexander, young in age but noted for his valour and swiftness of action, on one wing, placing beside him his most seasoned generals, while he himself at the head of picked men exercised the command over the other; individual units were stationed where the occasion required. On the other side, dividing the line according to nationality, the Athenians assigned one wing to the Boeotians and kept command of the other themselves. Once joined, the battle was hotly contested for a long time and many fell on both sides, so that for a while the struggle permitted hopes of victory to both.

Then Alexander, his heart set on showing his father his prowess and yielding to none in will to win, ably seconded by his men, first succeeded in rupturing the solid front of the enemy line and striking down many he bore heavily on the troops opposite him. As the same success was won by his companions, gaps in the front were constantly opened. Corpses piled up, until finally Alexander forced his way through the line and put his opponents to flight. Then the king also in person advanced, well in front and not conceding credit for the victory even to Alexander; he first forced back the troops stationed before him and then by compelling them to flee became the man responsible for the victory. More than a thousand Athenians fell in the battle and no less than two thousand were captured. Likewise, many of the Boeotians were killed and not a few taken prisoners. After the battle Philip raised a trophy of victory, yielded the dead for burial, gave sacrifices to the gods for victory, and rewarded according to their deserts those of his men who had distinguished themselves."

This, in addition to a comment by Plutarch about the Theben Sacred Band being undefeated until this battle, is about the sum total of our information, not including what you might recover from archeology.

You might have noticed that these don't even sound like they are describing the same battle. This is an additional problem with descriptions of ancient warfare in that many of the descriptions of them are incredibly suspect without even suspecting the authors of bias. Warfare is incredibly chaotic and thousand soldiers are going to tell a thousand stories of how that battle went. What if only one of these fragments had survived?

So in the end we are left with a huge degree of speculation. We know that when they met, greek hoplites lost to a macedonian phalanx. But why? Was it because the Macedodian arms were superior? Was it that their training allowed them to outlast the initial Greek attack and exploit the amateur athenians into making a tactical blunder? Was it because Alexander succeeded in breaking through? We are not even sure if Alexander's contingent attacked on foot or as cavalry. We are also not sure if Philip's contingent that apparently intentionally gave ground were the famous hyspaspists and if they fought as phalangites or heavy skirmishers or both.

So, phalangites replaced hoplites because phalangite armies beat hoplite armies, or possibly more accurately:a phalangite army beat a hoplite army and that was the only time that it mattered in the end.

Sources: Armies of the Macedonian and Punic Wars: 359 BC to 146 BC, Duncan Head Polyainos IV.ii. 2&7 Diodoros XVI.86 Plutarch: Pelopidas XVIII