This question is inspired by the question Why did pila fall out of favor for infantry? which was posted last month by /u/paladin_ranger. The question went unanswered but sparked a small discussion here.
Introduction
Early on in Roman history the pilum (javelin) became one of the iconic weapons of the Roman legionary. By all accounts the way the Romans used them - throwing them at a short distance, before engaging with their sword - was pretty effective. By the time of Vegetius however, the pilum was hardly used anymore. Vegetius says:
As to the missile weapons of the infantry, they were javelins headed with a triangular sharp iron, eleven inches or a foot long, and were called piles [pila - edXc.]. When once fixed in the shield it was impossible to draw them out, and when thrown with force and skill, they penetrated the cuirass without difficulty. At present they are seldom used by us, but are the principal weapon of the barbarian heavy-armed foot. They are called bebrae, and every man carries two or three of them to battle. (Vegetius I.XX, emphasis mine)
So by the time of Vegetius the pilum appeared to be phased out, possibly in favour of the plumbata:
The exercise of the loaded javelins [Plumbatarum - edXc.], called martiobarbuli, must not be omitted. We formerly had two legions in lllyricum, consisting of six thousand men each, which from their extraordinary dexterity and skill in the use of these weapons were discingui.shed by the same appellation. They supported for a long time the weight of all the wars and distinguished themselves so remarkably that the Emperors Diocletian and Maximian on their accession honored them with the titles of Jovian and Herculean and preferred them before all the other legions. Every soldier carries five of these javelins in the hollow of his shield. And thus the legionary soldiers seem to supply the place of archers, for they wound both the men and horses of the enemy before they come within reach of the common missile weapons. (Vegetius I.XVII)
(Thanks /u/Celebreth for the pointer to this quote.)
Actual question(s)
What do we know of the switch from the pilum to the plumbata? Why and when did the pilum, the iconic legionary weapon, fall out of favour for Roman infantry?
The decline of the pilum within the Roman military likely began as a result of the Crisis of the Third Century and subsequent events that followed. Many changes within the Roman legions resulted from the decline of a central and consistent military authority within the Empire starting with the death of Commodus in AD 192 - civil unrest and ever-present rebellions ensured that the various arms of the military swore oaths to wealthy citizens or generals rather than to the state or the Emperor, eventually changing the nature of the legion itself. In the end, the Roman reaction to this disastrous and devastating century was what hammered the final nail in the coffin for the pilum.
Following the restoration of the Empire begun by Aurelian in AD 274 (with the defeat of the Palmyrene and Gallic usurpers), which was was later solidified by Diocletian in the 280s, the Roman legions, the backbone of the Empire, were once again restored to a state of relative order for the first time in nearly 60 years. However, keen on eliminating the inherent weakness in the Empire's system of leadership, Diocletian, as part of his reforms, sought to permanently change the way in which the military operated.
First and foremost, Diocletian cemented the idea that the Roman legions were no longer the massive, provincial armies tasked with fixing roads, building bridges, launching unexpected expeditions to conquer far-off lands, and most importantly, operating outside of the Emperor's direct control. After what had happened in the previous decades rife with political assassinations and rebellions, Diocletian would not have anything less than direct control. His armies would be loyal to him, and to him alone. But the problem was, Diocletian knew that one man could not control all of the legions, nor the whole of the Empire, which is what led him to design the Tetrarchy. He also knew that the legions had to be rebuilt, reorganized to fit a rapidly-changing, and more dire era of Roman history than that of Augustus or the Flavians or the Antonines.
As a military man, Diocletian felt that assuming command of the army, and holding his soldiers' unswerving loyalty, was the key to survival. He therefore believed that in order to secure his Empire, a mobile army, essentially a grand retinue that followed him wherever he would will to go, was what would ultimately provide the best security ("might makes right!"). With all of the corruption and rebellions that had occurred in years past, this was an extremely sensible and wise decision. Wherever the Emperor went, the army went, and this also held for the Augusti and the Caesares of the new Tetrarchy system. The capitals of Diocletian's new system (Sirmium, Nicomedia, Trier, and Mediolanum) can be seen, in essence, as headquarters from which to resupply the Imperial retinues for campaign.
But, how did all of this affect the decline of the pilum, you ask?
Not only were these changes helpful for the mobility of the armies, but they were also more economical. The new weaponry was by-and-large cheaper to produce, and much easier to repair - helpful changes for an Empire whose economy had been driven into the ground by internal strife and (later on) devastating invasions. The fact that the pilum required a large amount of wood, iron, and lead to produce would have been seen as a serious drawback, since it was essentially a single-use weapon. The martiobarbulion, on the other hand, was much smaller than the pilum, and required little more than an arrow's worth of materials.
But perhaps most importantly, the pilum suffered from poor performance when used on horseback, which limited its use in the increasingly cavalry-dependent armies of Late Antiquity. This was especially true in the Eastern half of the Empire, where wars with the Sassanids essentially required Roman armies to have large contingents of either ranged or armored cavalry. Many plumbata could be carried in a saddle bag, where pila would not have been easily carried at all. Barbarian armies that threatened the Empire in the Late 4th and 5th Centuries (such as the Huns and the Visigoths) almost universally made heavy use of their cavalry, which again, pushed the Romans to adapt their field armies towards mobile cavalry forces than infantry.
But what I am ultimately getting at is that the plumbata/martiobarbuli had significantly greater range than the pilum, which allowed the legions far greater flexibility in combat against their foes. Again, to drive the economic points of the matter home, as Vegetius remarks, legionnaires armed with plumbata could "take the place of archers", allowing them to fill the shoes of essentially two units with one. Additionally, having personal experience with throwing martiobarbuli, they are essentially self-stabilizing, meaning that they required limited training to use, which is in direct contrast to the pilum, which required extensive training to throw for any useful distance. Since Vegetius remarks that the quality of the legionary training had declined, it makes perfect sense why Roman generals would want to adopt this weapon.
In conclusion, the reason why the martiobarbulion/plumbatum eclipsed the pilum was, quite simply, because the former was a superior weapon in almost every way. Not only was it smaller, lighter, and cheaper to produce, but it also required limited training to use effectively, could be thrown from horseback, could be thrown in greater volleys/numbers than the pilum, and had a vastly greater range that allowed regular armored infantry to engage an enemy at long range. I really can't think of a weapon more perfectly suited for its time.