During the time period it was established, it faced up against enemy musketeers, who obviously had a range advantage over the half-pikeman unit. So why did it work?
the idea of the tercio was that it was a sort of jack of all trades formation.
The pikemen in the ranks made it so that cavalry couldn't just run up to the musketmen and cut them down, thereby increasing the effective time the musketmen were firing. the mixed ranks also made it so that there wouldnt be a weak spot in the lines due to unit composition, however poor positioning could still create a weak spot in a battle line.
The Pikemen also stopped full pike/melee formations from closing the distance and chopping the musketmen to bits. The pikemen keeping the enemys away from the musketmen allowed them to be more effective, as there was less of a chance of being cut to bits and thus they would have been less panicked.
The Tercios only real weakness was full musket formations (somewhat) and also artilley. against full isolated musket formations(no surrounding pikemen/melee) the tercio could close the distance and then use their pikes to force the enemy formation to break. but if there was supporting pike/melee or cavalry units, then closing the distance would be most likely suicide unless the opposing force was Heavily demoralized.
Artillery however, could do significant damage to a tercio as it is a very dense formation, and artillery is favored against infantry in general but especially against tightly clustered infantry. While the tercio could and likely would go into a looser formation if faced against artillery, avoiding the Artillery fire completely would have been favored and thus they would have manuvered out of the fire zone of the Artillery, if possible.
A reply to /u/arsonistSnowman
The tercio was a close-order formation where pike & shot sub-units collaborate and adapt depending on the threat they faced.
How do they adapt? The answer is that different elements of a tercio -- which itself is an organizational term, not a battle formation -- will move relative to each other depending on the threat.
Here is a great illustration of how elements of a pike&shot formation adjust to a threat from its right. In this case, they maximized the frontage of missile troops to pour fire onto an incoming enemy.
Further to quote F. G. de Leon, The Road to Rocroi: Class, Culture and Command in the Spanish Army of Flanders, Brill (2009), it was the Sergeant Major who was central to "almost all of the army's decisions," as he was responsible to train the soldiers in various formations and maneuvers, and in battle he was the one to receive orders from the Maestre de Campo or Colonel, and send them to company Sergeants who lead their companies in battle.
So basically, the "perfect rectangles and squares" that you see in many illustrations were not completely fixed in battle.
If you consider later periods into the mid 1700s, pikes started to play a strictly secondary role to muskets, as seen here, an excellent cross-section showing the interaction between pikes and muskets. Further, it is possible to combine the pikes into one massive grouping when it is facing another pike-heavy formation or cavalry.
When a mixed tercio faces a purely musket formation, it would rapidly advance with the pikes while their shot elements will provide punishing fire. The shot squadrons or brigades would be placed on the sides, such that when the pikes make contact with the enemy, they could fire in enfillade.
Battles in the early modern era were most often affairs settled in close combat, this is where the mixed tercio formation excels.
Facing a musket formation? Push with the pikes while the shots punish the enemy continuously.
Facing a cavalry formation? Fire off a shot or two to punish them while they advance, then pull the shots back behind the pikes.
Facing a pike formation? Use friendly pikes to stabilize the front while our shots pour fire into their pikes.
So you can see how the tercio was very flexible and really hard to beat. That's why they were widely copied.
Lopez, Ignacio (2012-07-24). The Spanish Tercios 1536-1704 (Men-at-Arms) (Kindle Locations 772-774). Osprey Publishing. Kindle Edition.
F. G. de Leon, The Road to Rocroi: Class, Culture and Command in the Spanish Army of Flanders, Brill (2009)