What benefits were there to the former that made such a massive transition worthwhile?
Soldiers armed with flintlock muskets and bayonets were what replaced infantry armed with matchlocks and pikemen. The short answer would be, of course, that they replaced them because troops with them tended to win the battles.
The advantages of muskets and bayonets was pretty obvious: Before, there was one soldier with a long-distance weapon ( matchlock) and another soldier with close-in weapon ( pike). . There were limitations with pikes. They were pretty good in simple defensive positions against cavalry, but in a block of pikemen only the front ranks did most of the effective work. To maximize their distance effect ( beyond sword range) pikes also tended to be too long for very close combat ( often around 16 feet) and two blocks of opposing pikemen could get pretty tangled up and so become bogged down. A musket with a bayonet replaced two soldiers: it gave one soldier both a long-distance weapon ( musket) and a close-in weapon( bayonet).
Matchlocks required care. Each soldier had a tin tube dangling from his waist to carry some match. That either slowly burned, or it had to be lit. Naturally, if the soldier was to be ready, the match had to be lit- and kept lit. That lit match was also never far away from the loose gunpowder the soldier would be using to load his gun, especially in priming the pan,and care had to be taken that there wasn't a slip and an accidental firing. The string of cartridges often seen hanging from bandoliers, in the late 16th c and earlier 17th c., could go off , leaving a very unhappy soldier. There was also the problem of weather. Black powder is hygroscopic, and if at all damp will not work. Muzzle-loaders of all kinds have problems with rain and humidity, but a flintlock can be at least partially shielded - grease put under the steel to seal the pan, the gun carried with the lock in the armpit. Not possible with lit match.
However the advantages, the change would happen pretty slowly. Matchlocks are simpler, and so, cheaper. Though the French would introduce the first practical flintlocks in the 1630's, and they would become more and more popular during the 30 Years War, matchlock muskets and pikemen would still be around into the 1690's, and even later. But pikemen by then would have been much fewer, and would long have been taken out of their blocks and mixed with soldiers with muskets
I assume you mean the rifle, and not the pistol
The answer is a development in tactics and firearms technology. Arquebuses used during the age of pike & shot were heavier, slower to reload, and would often be placed in y-shaped holders to steady it during firing. This, and their comparatively shorter length to muskets, made gunners vunerable to cavalry charges. Thus, they were protected by pikemen who could repel cavalry charges but were themselves vunerable to arquebuses.
When the longer and faster to reload muskets were introduced, more and more arquebuses were already being added to infantry formations as the advantages of superior firepower on the battlefield were greater than more meele fighters. The Swedes under Gustavus Aldolphus removed the halberd fromt their formations and made the infantry entirely equipped with pikes and guns.
A musket has a superior rate of fire to an arquebus, and increased firepower leaves the musket as the superior weapon of choice instead of pikes for every soldier. Second, a musket with a bayonet is essentially a spear, giving every man a polearm to fend off cavalry. Suddenly your entire formation can both shoot, fend off cavalry, and fight in meele if needed.
So the superior firepower and increased flexibility of a musket with a bayonet will outperform arquebus and pike nearly every time.