I have to say I think you're taking too much from Total War game and apply it to the battlefield.

But considering cavalry charges often break apart and fail at the first row of pike-armed troops this brings another point

I'd need examples of cavalry charge being stopped by one single rank of pikemen. In fact, I need many.

In particular the sources say that sword and shield men often not only counter but easily defeat entire units of pike and spearmen

Not true.

Landskricht had shock troops armed with heavy two handed swords designed to cut off the sharp tips of spears and pikes to render them uselesss.

You seem to be forgetting that the Landskricht halberdiers and zweihänders and the Spanish sword-buckler of the tercio were there in small numbers for support. Most of the fighting was still done by the pikemen. If pike squares were as easy as you say to break through, they'd be pointless. Also, halberds are polearms as well.

The Japanese also employed a similar tactic with their NoDachi swords (less cutting off the pole apart and more parry in and kill an individual ashigari or Yari samurai).

Not a single surviving mobilization record (that I have seen anyway) mentions the Nodachi. This mean that the Nodachi, like many other weapons, were deemed as "free-equipped". A man brought it and used it because he liked it, not because anyone ordered him to. Therefore, there's no way the nodachi was used in an assigned way to break apart pike formations.

Not to mention there was no such thing as pike formations in Japan. There were only formations that had pikes. And there's no such thing as yari samurai either. If you were a full-status-no-questions-asked samurai, you'd most likely go into battle with full armor, a yari, and your two swords, and maybe a horse. But you really went with whatever you want.

Hell you don't even need troops designed to counter spearmen to beat them- you can even get lightly armed soldiers such as random militia armed with heavy clubs and so long as a few units they don't rout and stand their ground, you can send other units who are not directly engaging to flank the spearmen (where they are vulnerable), rush in before they realize the flanking, and kill them before they grip their spear underhand or overhead.

I doubt any untrained, barely-armed, and incredibly lightly armored militia would stand their ground against anything for very long.

I know some soldiers such as the Spartans and Macedonians had mastered using pole arms so well that the second man in row can easily do an overhead attack over the first row of spearmen to aid in killing the enemy (as the Spartans frequently did) and int he case of the Macedonians, the front wars had ways of utilizing several rows at once.

Just a correction here. The sarissa of the Macedonian Phalanx was held underhand, and their length allow 4 points to extend in front of the first man.

man in front crouches while the man behind holds spears in a straight row and the third row angle it upwards when awaiting a charging enemy

I believe this is a brace stance to receive cavalry used in early modern Europe.

But this still goes against logic why you need to arm an entire division with nothing but pikes since there are so many weaknesses.

It is fairly simple to duck or slide aside one sharp point. It is much harder to go through four or five points. But that's what you have to do against pike blocks before you can get at the other guy, unless you also have a pike. In early modern Europe a square shape also allows the formation to quickly change direction (four directions) or form into a sort of hedgehog if surrounded.

A lot of the rest is for morale, whether pike or swords. Men in deeper ranks are less likely to run away.

Or even have some of the spearmen in the second row drop their spear and pull out daggers or sords to exploid the pike's weaknesses.

This was one of the many ways some generals in early modern Europe advocate piked blocks be used.

Since cavalry typically are stopped easily by the first row of pikemen in a direct charge, couldn't mercenaries simply have a unit in the aforementioned manner above (spearmen in front, swordsmen and/or everything else behind)? I mean this setup makes more sense than entire units since spearmen already have proven to be very weak against sword and shield and two handed sword troops!

Sure. If the cavalry are easily stopped by a single rank of pike. Which is probably why you find something kind of like what you advocate for in Japan where the horses are few in number and unsuited to charging.

But in mainland Eurasia the charge seemed to be much more effective with either stronger horses in Europe, or charging against peasant militias who had a tendency to run in China. A cavalry charge is scary. A battlefield is already very scary. When you see a huge beast charging at you with a rider on his back pointing a weapon at you, your instinct is to get out of the way, because even if the horse just crash into you, you'd be in a world of hurt. IIRC there are a few examples here and there of pre-Alexander cavalry straight up breaking a hoplite phalanx with a charge. These are not even all that heavy cavalry with all that large horses, and hoplite phalanxes are usually 4~12 deep IIRC. They could usually stand their ground, but that we can even find exceptions show how scary it was.

But if you do, then the cavalry charge has succeeded and your formation would be cut into pieces. Extra depth in the form of more bodies not only means more pointy sticks are in front of you to make sure the enemy, horse or men, don't get through, but also that you can't physically run away. You'd of course also feel safer with lots of other comrades all around you.

It's important to regard battle as a psychological game, not what you could physically do with a weapon.

As u/ParallelPain points out, it often did take more ranks of pikes to stop a charging horse and rider than you might expect. There are also other benefits to having multiple ranks in that it makes it more difficult for the enemy breaks through and adds more "mass" to the line (even if this was likely a psychological factor for the most part). As a result 4-8 ranks tended to be pretty common for medieval infantry depending on the number of troops available.

What I suspect you're confused about however are the "great squares" of pikemen which dominated European warfare in the late 15th through the 16th century. These pike squares could seemingly get absurdly deep, up to 50 ranks, and were sometimes even longer than they were wide. These tactics were pioneered by the Swiss and essentially a response to the difficulty of command and control when it came to medieval armies and the need to turn infantry into an effective offensive tool.

In earlier times, attacking infantry was almost always at a major disadvantage since it was almost impossible for a long, thin line to advance without losing its cohesion. What the Swiss found was that well-trained pikemen in a square or column had a much narrower frontage and could make rapid assaults without losing formation. Then once they reached the enemy, the much deeper square would have no problem driving back the thinner enemy formation. Medieval battlefields were notorious for the difficulty of communication and when infantry were arranged in a long, thin line it became impossible for troops in one place to know what exactly was going on over in the distance. As a result, once the center of the line started to be driven back it typically began a domino effect which caused the entire army to start routing.

In the hands of the Swiss these aggressive tactics were extremely effective and during the 15th century swiss pikemen proved capable of defeating much larger combined-arms armies which included far superior numbers of archers, handgunners, and cavalry. And while other armies typically didn't quite match the discipline of the Swiss, they did copy the tactic of large pike squares to good effect.

As for the dominance of the pike. The main source for the idea that pikemen were weak against sword and shield troops seems to be Machiavelli's the Art of War. While it was a very influential work, Machiavelli was not a soldier and in general his dream of a return to Roman-style sword and shield infantry does not seem to have been taken that seriously. The general opinion, on the contrary, was that a well formed pike square was almost impossible to attack head-on with anything but another well-formed pike square, and even among the Spanish the trend became fielding more and more pikemen at the expense of all other sorts of melee infantry. The advantage of the shorter weapons like swords, bills, and halberds is described as being when fighting on rough, confining ground such as a trench or breach, or when the formations have otherwise lost cohesion, such as what happened when english billmen defeated the scottish pikes on the slippery slopes of Flodden. During the Italian Wars the Spanish typically relied on breaking up the enemy's charge using ditches and volleys of arquebus fire before the infantry counter charged. The famous "Rodeleros" weren't really relied on to break up pike squares on their own.

As a slight tangent, Even in these scenarios it might be more accurate to say that the ideal was more a mixture of pikes and short weapons rather than pure targeteers. The way Sir Roger Williams described it, when attacking into a breach, it's still the pike points that lead the push, while billmen, halberders, and targeteers back them up and defend them. Fencing writer George Silver actually considered the pike a pretty effective weapon in one on one combat and even gave it the advantage over sword and target in that scenario.

A 16th century pike square which incorporated significant numbers of shorter weapons, would often look somewhat like this English Example, where the Bs in the center represent billmen, halberders, or targeteers surrounding the colors, chief officers, and musicians. When a "push of pike" occurs it is the heavily armored pikemen in the front rows which do most of the fighting while the shorter weapons mainly just add mass. Once the enemy formation starts to break apart, then the shorter weapons can move out of the square in order to pursue and execute as many as they can. Alternatively if their own pike square breaks apart then the shorter weapons are theoretically more effective at protecting the colors and officers as they retreat.

I say theoretically because as I mentioned the trend was for armies to include fewer and fewer numbers short weapons until during the 17th century halberds and the like became almost strictly carried by officers. According to Robert Barret, the practice of mixing shorter weapons in with pikes typically proved useless since whenever the front ranks of pikemen were repulsed, they would step backwards into the ranks behind them and the press would be so great it became impossible to use even a short sword. He recommended as few troops armed with halberds or sword and target as possible and suggested that they would be more effective mixed in with the gunmen in the flanks and the forlorn hope. However the decline in use short weapons may also have had to do with the fact that by the end of the 16th century tactics were changing yet again. The "great squares" were being replaced by tactics which involved larger numbers of smaller pike squares arranged in a mutually suporrtive checkerboard-like formation, which combined with improved organization made armies far less susceptible to mass routs. And melee weapons in general were increasingly playing second fiddle to firearms. It might be that the shorter weapons were being used less because it was no longer fruitful to break formation to pursue a single enemy battalion, and the light infantry duties previously performed by billmen or sword and shield men were now being much more effectively performed by musketeers alone.

Rogers. European Warfare, 1350-1750

Hall. Weapons and Warfare in Renaissance Europe: Gunpowder, Technology, and Tactics

Webb. Elizabethan Military Science: the Books and the Practice

Taylor. The Art of War in Italy, 1494-1529

Parker. The Cambridge Illustrated History of Warfare

Silver. Paradoxes of Defence (1599)

Robert Barret. The Theorike and Practike of Moderne VVarres (1598)