Specifically how were the Romans with their gladius and pila able to get close to the Phalanx with their long Dory spears. I have heard it has something to do with the maniple formation but don't understand how that worked in practice.
Initially, you're right; like anyone else who tried to take on a phalanx head on, the Romans under Titus Flamininus were initially repulsed at Cynoscephalae (198 BCE). The attack was so silly, trying to go uphill straight at Philip V's right, that some historians have speculated that it might have been a feint. In any case, Philip's soldiers pursued the Roman left's retreat onto broken, hilly terrain. This opened up a gap between Philip's right and center, which Flamininus exploited with elephants he'd acquired from Antiochus III. The Romans exploited the gap to maneuver freely around the Macedonians and encircle them. The phalanx, naturally, is incredibly difficult to "turn", takes time to establish, and is terrible at adjusting to things like broken terrain or sudden changes in battle. If the enemy takes it on, like Darius III, it's amazing. If an enemy can get advantage with skirmishers, or the legionary pila, or flank the phalanx, it's finished if its cavalry are neutralized.
Lucius Aemilius Paulus had similar initial trouble at the Battle of Pydna, in 168 BCE, against Philip's son Perseus. His men tried cutting off the heads of the Macedonian pikes but got pushed back and were really getting whipped. Someone even threw an eagle into the Macedonian line to inspire the men to attack and get it back, which failed. Once again, the Romans retreated to uneven ground as they were pushed back. Once again, the phalanx lost its cohesion, which Paullus saw. He ordered the legionaries to attack the gaps, and they got in close since the phalanx was disorganized. In close quarters combat, the Roman gladius was longer than the Macedonian short swords, Roman armor was superior, and the Romans certainly practiced the stabbing motion of the gladius more than the phalangites. Perseus realized things were falling apart and fled with his cavalry before even engaging with the Romans. This was the end of Macedon, and Alexander's empire (save for the Ptolemies).
Probably the worst thing that ever happened to Macedon was, ironically, its ally Hannibal. Philip V allied with Carthage during Hannibal's Italian campaign, only to see him eventually lose and leave Philip V alone vs. Rome. Furthermore, although the legion had adopted the manipular "phalanx with joints," Roman tactics still centered around a massive central push to break the middle of the enemy army, protected by Socii allies and cavalry on the flanks. Hannibal diverted the cavalry and surrounded the legion at Cannae, leading Africanus, Fabius Cunctator, Flamininus and others to reform the system and rework tactics to avoid another Cannae. The old system would have played directly into Macedonian hands.
Beyond that, the Romans were good at playing off the Aetolian and Achaean leagues and using "divide and conquer" politics to conquer Greece, although the various Greek states assisted with their tendencies toward internecine rivalries.
Flamininus has a section in: Plutarch. Lives of the noble Grecians and Romans, edited by Arthur Hugh Clough. New York: Benediction Classics, 2010.
Also good reading at: Hammond, N.G.L. "The Campaign and Battle of Cynoscephalae." Journal of Hellenic Studies 108, 1998.
Montagu, John Drogo. Greek and Roman Warfare: Battles, Tactics, and Trickery. London: Greenhill Books, 2006.
Were the Romans using the gladius when they conquered the Greek city state? I thought it came along later, after they got involved in Spain.
A side question if it is allowed.
What would have been the affect of the pila on the Phalanx? If the pila was thrown head on with no melee action?
Would it have been effective at taking about the shields of the hoplites? Would it have mattered once things did reach a melee?