Because real war is not rock-paper-scissors. Battles are far more organic and it is seldom a case of a clear-cut "counter" to another unit. One has to consider all the factors - the training, morale, coordination and leadership of the troops; the environment and terrain; and the myriad factors that get thrown into any conflict. It's not as straightforward as being a time commander and right-clicking the cavalry to move towards the missile units.

Another issue that makes this topic difficult is that in many cases, the weaknesses were exploited and the "elite" units were defeated. The thread you linked is an absolute mess. Broad generalisations, few specific examples and vague references to "elite" units.

Let's go through some of these supposed counters individually to see where it breaks down.

Indeed like the OP said despite the Spartan Phalanx having the weakness of the right flank exposed, few city states dared faced Sparta in its prime out of few and those that did could not attack the right flank for whatever reason.

Firstly, the right flank isn't uniquely a weakness of the Spartan phalanx; that was the vulnerability of all Greek phalanx armies, and it was normal for these formations to drift to the right since individuals would seek protection from their neighbour's shield. Because of this opening, it was the norm for Greek armies to stack their elite soldiers on the right flank. Thus, in theory, while it was the most vulnerable side, it also had the best soldiers who were least likely to break. Thus, it might come down to whose left flank would collapse first.

If you are imagining that one simply has to encircle the right flank, the formation would simply spread its ranks to match the enemy line before engaging. The movement of light troops (skirmishers, cavalry) would prevent wide sweeping movements. Armies were seldom so lopsided that one could simply walk around the right flank and stab.

The exception to this would be the Thebans under Epameinondas, who stacked his left flank to counter the Spartan right at the battle of Leuctra. But this was a gambit, and flies in the face of the military and cultural expectations of putting your best troops on the right.

An even more consistent example is heavy cavalry.

It's not clear exactly which period you are referring to. This is so broad that as much as I want to dig into this, I really can't.

I will say that it's essential not to treat the matchups of cavalry versus pikes in isolation. The most important element of the army is the footman. The spears and shields that make the line of battle are the ones who carry the fight. The cavalry are opportunistic and are used when they are most effective. While pikes could defeat cavalry, pikes were vulnerable to archers, which is what happened to the Scottish schiltrons at Falkirk, Hamildon Hill and others.

The pattern you'll see is that for every "counter" you have to something, it too has a "counter". But the other pattern is that as much you might perceive that elite units aren't being successfully "countered" with these weapons and tactics, in many cases they do exactly that.

I am curious why despite learning weaknesses and having the tools to exploit them, elite units are still extremely difficult to battle and can even do curbstomping battles against forces attempting to use the weakpoints against the elite units (such as long pikes against knights)?

Again, it helps if you refer to specific examples of "curbstomping", as often the context will reveal why specific armies were able to win the day. As for the pikes, as I said above, armies were not equipped and trained to fight against knights specifically. They had to fight against the other army, including their spears and archers. And from a logical point of view, a knight probably isn't going to charge into a formation they wouldn't think was easily overrun.

What is it about real war where its not enough to position troops on the high ground and shoot musket rifles at very slow heavy infantry trying to walk up the mountain to defeat your army

In most cases, if you have the high ground and you're going to massacre the enemy army, the enemy commander is going to refuse the battle and wait until you come off the hill, or circle around the high ground to threaten your supply line and find a way to regain an advantage. Commanders would obviously exploit every advantage they can, and there are certainly many examples of commanders being on the defense and enticing the enemy to attack.

Not to mention that lining up your infantry on a hill to shoot down means you're equally exposed to enemy artillery. Hence the pattern in this response: every move has a weakness that can be exploited.

Or lightly armed archers defeating macemen in melee despite archers being weak in CQC because of minimal training and less armor as well as shorter and weaker hand to hand weapons?

We're thinking of this too much in gaming terms (which is where I think you're coming from). Archers are actually not weak in the melee. While they might lack the armour to stand up in a prolonged fight, a typical archer wasn't just a ranged fighter; they were also trained soldiers. They would have been equipped with swords and trained in fighting as much as any other soldier. Again, depending on exactly which period you are referring to, the "lightly armed archer" might be exactly the same as a spearman with the same armour plus a bow, or they could be a samurai, or they could be an Englishman at Agincourt who could charge in with their swords and mallets as light infantry.

Overall, you can't really look at military as rock-paper-scissors. It doesn't work that way in real life when you consider command and tactics. Not to mention that the logical "counters" in real life can counter each other. The obvious way to defeat archers would be to charge them with cavalry. How to defeat a cavalry charge? Shoot lots of arrows at them. Ultimately, the army that is better prepared to adapt to the battle will prevail, not the specific composition of "elite" units.