Basing this on Jonathan Sumption's work on the Hundred Years' War, which doesn't cover the entire time period you're talking about but I think still works. This is more for the 14th century in western Europe.
It's true that massed pikes were often very effective against the heavy cavalry charges that characterized the fighting style of a lot of the nobility in this period, especially in France. Provided that your formation had the discipline to maintain itself against such a charge (no sure thing). The issue was that the densely packed formations required to achieve this goal left the soldiers very vulnerable to massed longbow fire, which was a superweapon in pitched battles for this period.
Take a battle like Halidon Hill, where the massed scottish schiltrons came up against English longbowmen. The scottish spearmen were indeed strongly defender against heavy cavalry, but the English didn't use a lot of heavy cavalry because they didn't have the same quality of horses as other cultures did (a large reason for their adoption of the longbow in the first place). Rather than charge the scottish fixed positions, the English simply advanced into longbow range and poured arrows into the packed spearmen. Since they could just continually do this, then the scots needed to take the initiative and charge the English positions. As they did so, their formations broke over the difficult terrain of the battlefield and the scots lost the battle handily. Broken formations for massed spearmen had been a problem for these formations going all the way back to greek phalanxes.
The longbow also turned out to be as effective against massed cavalry charges as the spear formations were. The horses were difficult to protect against arrows. They'd get wounded, panic, the massed charges would break up, the heavily armored riders would fall when their horses were shot out from underneath them. Once the knights lost their mounts then they were sitting ducks. So you didn't need spearmen to deal with cavalry if you had longbows. Now, this is a big "if", since it's fair to say that it's easier to train spearmen, but if you could get longbows then you wanted longbows.
Fast forward a few decades into the hundred years war and you no longer saw these massed cavalry charges in most situations in France. Cavalry was used more for strategic mobility than tactical combat, as soldiers tended to dismount to fight. Cavalry dominated in the sense that horses move faster than humans and so you need that mobility to achieve strategic goals. But in a tactical battlefield, cavalry was largely ineffective against well trained bowmen.
Tl dr; In the 13th and 14th centuries the longbow showed that it was effective against heavy cavalry and against massed spear formations in the war between france and england.
u/beatsieboyz has provided a very good explanation on the evolution of pikemen during what we can call their adolescent years. In fact, from the 1400s and 1500s onwards, pikes start gaining (or re-gaining) primacy on the battlefield at the expenses of other types of troops.
As an additional explanation and answer to your query, the moment when these pikemen were able to begin to equip and organize themselves in different manners was the same time when the social and economic structure which bore the supremacy of the aristocratic heavy cavalry had undergone severe changes.
Up until the second half of the XIII century, in many instances across Europe, military service of the upper classes was required, based on a nebulous aggregation of oaths, obligations, customs and the like. This is what historians call "vassalage" or "vassallatic-beneficiary system", or "feudalism" if you wish to cause a stroke in someone, and did include the possibility for an individual of high social condition and wealth to call to his aid retainers and loyal subjects. This action meant to be able to amass a group of people most likely of high economic status, able to afford the best equipment for warfare that money could buy: a horse, metal armour and weapons.
This, alongside the comparably less effective employment of masses of poorer soldiers, armed with spears, shields, bows or clubs, as a Carolingian source states, the Capitulary of Aachen (802-803) about men having to bring at least a bow and a quiver, rather than coming armed only with a baculum, a cudgel, if they were too poor and couldn't be equipped by a wealthier lord, meant that for a long time, mounted warriors were preferred. This was also the case in places with a stronger urban presence like Northern and Center Italy, where cities like Milan, Pavia, Cremona and Florence had wealthy families of notaries, merchants and jurists which began buying horses and armour, becoming known centuries later as knightly families.
This cavalry was a virtual guarantee of quality troops, as very little other people could afford to train all their lives to fight on horseback, aside than affording the equipment. Even if they weren't a standing army, nobles were relatively professional warriors, seeing warfare as a mixture of fun and employment (as it was common to capture and not kill an enemy aristocrat, partly due to social norms and partly for the ransom which would be brokered). Even though virtually every able bodied man could be called to war, it was considered not as effective to field a hundred spearmen drafted from farms rather than two dozen knights. This is one reason contributing to the low numbers of combatants in Medieval armies.
We can also discuss that the common person, the average peasant or city dweller of the Middle Ages, didn't fancy war very much. Taking into consideration the ideal distinction of such society as portrayed by Adalberon of Laon, consisting of oratores, bellatores, laboratores, "those who pray (the clergy)", "those who fight (the nobles, the knights)", and "those who work", it was indeed true that only one of these social groups actively and willingly took part in warfare. We can make an example with sport competitions: technically, anyone can join the next judo competition in your town. But maybe you won't go, because you've never trained judo or physically. You neighbour next door, who's been a practitioner for 14 years, is eager to join. Dino Compagni (1246-1324), a merchant and chronicler of Florence, tells us that nobles were those who wished to draw their swords and get in a fight, where instead merchants and others (which he intends as shopkeepers, craftsmen, bankers, lawyers and the like) would have settled the same matter through discussion and diplomacy.
beatsieboyz has shown how in later centuries the role of cavalry changed from more strategic mobility rather than tactical charge (note: tactical did still happen all the way to Napoleonic wars with lancers and cuirassiers, but only successful when infantry was not 'braced').
But if you look at other posts on Askhistorians posts you notice the change in the type of warfare from the earlier medieval warfare to the late medieval period. The earlier medieval period of warfare was characterized by a focus on raids and sieges.
In a raid or chevauchée, the cavalry would go around burning the land and farms of an enemy. Raids allowed an attacking force to feed their armies on the tax base of defending armies. Having a large static army of pikemen wasn't as useful because they couldn't catch knights on horseback. If the pikemen separated to speed up they risked being defeated in detail. Battles were skirmishes as the smaller attacking cavalry sought to fight off defending cavalry and successfully disengage.
Tl dr; in raiding type warfare it is better to have a smaller more mobile force than a larger slower one.
The other common in warfare is the siege. In this, a defending force keeps their army at a defensive castle forcing an attacking force to keep watch because it is still an "army in being" and they can sally forth to attack the rear lines of an army that ignores them. By making themselves an effective nuisance the attacking force must split their armies. In this context, the range at which the castle force can be a nuisance is larger for cavalry greater than pikemen.
Tl dr; even in sieges, the mobility of cavalry was still useful in their ability to harass distant armies.
while Pikemen are more cost-effective than heavy cavalry in PITCHED BATTLES. But until the development of the cannon, there was no incentive to give pitched battles. A heavy cavalry can form most military duties even if they were not the most cost-effective or optimized for it., the same cannot be said for pikemen. Commanders knew this and consequently placed more emphasis on having more heavy cavalry.
ConteCorvo explains the social aspects best, "In the early and high medieval periods, rulers did not have systems to pick between pikemen or knights they took whatever force they could."