A compact group of men with lots of pointy sticks is very difficult to attack frontally, and before rapidly reloading firearms were developed, was the most popular way of fighting. You might be familiar with the ancient Greek phalanx as used by the Macedonians. A compact spear formation is what Alexander used to create his empire, even if as a consequence of compactness it is very difficult to maneuver (the phalanx is easy to flank, which is why you need a lot of spears as well as a degree of combined arms for a Greek phalanx to be effective).

The prevalence of Pike weapons in the early modern period isn't so shocking because of their sudden appearance (pole-arms had always been around, their evolution into pikes and halberds and their dozens of variations isn't particularly shocking), but their effectiveness unsupported as disciplined professional soldiers started to use them is, relegating cavalry to a much less decisive role in warfare.

Unsupported infantry had repulsed cavalry in the past on multiple occasions. However, none had a better reputation for doing so consistently than Swiss Pikemen, who had been repelling Austrian incursions for a long while as well as repelling Brugundians. The difficulty to invade the Cantons that formed the Old Swiss Confederacy and their system of local militias organized to defend themselves not only certified Swiss Independence, but also made it such that poor men who knew little but how to fight (and do it well) were soon in demand in France and Italy. The Valois kings of France were especially dependent on the discipline of the Swiss mercenaries, who would form the center of the French line and spearhead the attack with a steady steamroll of pointy sticks.

The demand for mercenaries was also fed by the early emergence of the state apparatus: as the government of the Italian Communes became governments of sovereign states and the King of France imposed himself on his feudal subjects, the state treasury became a more rapid and efficient way of raising an army as opposed to armies composed of men who owned arms and armor as a consequence of land ownership and fought out of nebulous feudal obligations.

Why didn't they use shields? Pikes are very long and heavy (so they don't shatter under a cavalry charge), they can only really be held with a two-handed grip.

Pike formations were designed to counter cavalry charges that were aimed at the artillery and weaker back field of combat, general HQ and what not. They were to brace and have the pikes secured and pointed at the incoming cavalry to either deter them or cause them massive casualties if they did charge.

They didn't use shields as they would disrupt the formations and spread the pike tips out, actually doing more harm than good. They weren't supposed to fight individually, although they were sometimes forced to, but for the most part were there to counter cavalry charges and as such didn't require a shield.