The javelin was a weapon peculiar to Mediterranean warfare, especially for Latin/Italian, Iberian and Celt-Iberian peoples (Sanz 21). The use of javelins (not always the pilum, later the spiculum and earlier a different type, but javelins are all basically the same in function) remained a mainstay of Roman infantry warfare for basically the entire Roman period. We keep digging them up even in the late Empire (Bishop & Coulston 200). Indeed, missile warfare became in some ways more prominent rather than less in the late Roman period, with Germanic groups also using javelins (such as the ango), and there was an increasing trend toward training recruits in various types of ranged warfare, also including archery and slinging (Rance 251-3). Javelins did not go away with the 'fall' of Rome either, and throwing spears (again, the Germanic ango, which was based on the pilum) pop up in the Merovingian period, though it appeared to fall out of use by the 7th century AD (Halsall 164-5). Archery became perhaps more significant in this period, as it certainly was later (166).

I see no reason why any great developments in warfare (tactical or technological) should be responsible for this gradual fading away of the Mediterranean sword-and-javelin style of fighting. In my view, fighting styles shifted over time for reasons as much to do with society and culture as with technology and tactics. Perhaps the intermittent disappearance and re-use of phalanx-like tactics at various stages of European history might fit this bill: it's not as if shield-and-spear ever stopped being an effective weapons combination.* I think it is perfectly explicable by the massive shift over time in the culture of Western Europe, and the role of the javelin could be in many ways readily filled by archers and the multi-skilled soldiers of the early Medieval period (166). There is a tendency by Westerners to view changes in and the relative effectiveness of different military tactics and fighting styles from a 'technological' standpoint. For instance many casual observers tend to obsess over the exact size and shape and capabilities of the gladius and scutum in explaining Roman success and ignore the fact that the Roman army in the Mid-Republic was a highly trained, highly motivated force with hugely superior manpower reserves to its enemies. I suppose the point I am making is that shifts in military equipment and fighting styles do not necessarily have to be tactical or 'Darwinian' in any way, but can reflect changes in societies, cultures and approaches over time that are not necessarily 'superior', just different.

*I am of course aware that the Romans beat the Macedonians and Seleucids, who had all-but-perfected the classical phalanx, but tactics, generalship and superior Roman resources are important factors in this. Libyan (and Carthaginian) spearmen performed usually very well, even against Roman legionaries, in the Punic Wars for instance. Later on the Romans began to re-adopt spear-based fighting styles once more, though the performance of the Mid to Late Republican Roman legions clearly demonstrate that the Mediterranean style of fighting could, well handled, deal with any enemy (though I'm not factoring the fluctuating dominance of cavalry warfare in the ancient to early modern periods in here). The reasons for the use and re-use of spear tactics are probably as much to do with the quality of recruits and training, and cultural shifts, as any 'tactical' necessity.

Sources:

M.C. Bishop & J.C.N Coulston, Roman Military Equipment: From the Punic Wars to the Fall of Rome (second edition), Oxford: Oxbow Books, 2006.

Guy Halsall, Warfare and Society in the Barbarian West, 450-900, Abingdon, Oxon: Routledge, 2003.

Philip Rance, "Battle" from The Cambridge History of Greek and Roman Warfare (Vol II), New York: Cambridge University Press, 2007.

Quesada Sanz, F. ‘Not so different: individual fighting techniques and small unit tactics of Roman and Iberian armies’ in P. Francois, P. Moret, S. Pere-Nogues (eds) L’Hellenisation en mediterranee occidentale au temps des guerres puniqes. Actes du Colloque International de Toulouse, 31 mars-2 avril 2005. Pallas 70 (2006) 1-25.

Roman weapons evolved across the empire's history, and the pilum wasn't exempt these processes. The general idea - a weapon with a long, narrow metal shank that would bend on impact, rendering it useless by the enemy and possibly tangling the shield of the person it hit - stayed relevant, but the form changed.

By the late empire, we see the pilum replaced / evolving into a barbed throwing darts that archaeologists usually call 'angons' (a Greek word used by a Byzantine author to describe the pilum-like javelins used by 6th century Frankish soldiers). This image shows some reproductions of an early angon style that was popular in both the late Roman army and among 'barbarian' groups outside the frontier.

We find a lot of these javelins in bog deposits, like Illerup Adal. At Illerup, at sometime around 200AD (and on several occasions after that), a bunch of weapons were dumped into a bog. We're not entirely sure why, but the general theory is that they're the armaments of a disarmed, defeated army. The weapons were found in sets, and it was clear that warriors were armed with (among other things) a spear for hand-to-hand fighting and an angon-like javelin. In Illerup, as in the Roman army, we're looking at an organized force whose warriors whose equipment was standardized, including the provision of everyone with a made-to-purpose javelin. That speaks of a particular kind of centralized political structure: someone in charge equipped the soldiers with these standardized, specialized weapons.

As Roman power began to spin out of control in the west, however, this centralized control started to change. The army became increasingly privatized, and the responsibility to equip and support it devolved onto local elites. Regular weapon factories shut down, and production seems to have in many cases shifted to local production. Sometimes these locally manufactured weapons were still very regular, but in England at least there was an increasing amount of diversity of style, particularly in the spears people used. It's like if the army today shut down, and were replaced with a private citizen militia that had to arm itself: you'd see a lot of military surplus at first, but as time went on things would get more and more chaotic. And in England, and to a lesser extent in France, the weapons did.

What that means for weaponry is this: the organized, specialized kits of shield, spear, javelin, etc start to disappear from the archaeology by the end of the fifth century. That doesn't mean they necessarily disappeared from the army - people may not have been buried with every weapon they owned. But I personally think the variety of weapons we see in the ground reflects the larger change in military structure we see by the end of the fifth century in the west: there's no organized army anymore, and individual members of the warrior militia have a lot more individuality and personalization in their equipment. This is when angons start to become very rare.

An angon of the style more commonly found in the 6-7th centuries.

If you ask me why angons - the descendants of the pilum - ultimately vanished, I would say it's a result of a long process of military de-specialization. Warriors continued to use javelins, but the spears that are common in the archaeology during this period are generally more multi-purpose weapons that could be thrown or used in the hand. Some of these spears still have the long, narrow shank of the pilum / angon, but replace the barb with a more versatile leaf-bladed or angular tip (for example - and there are more extreme cases with a much longer shank). These multi-function spearheads (either singly, or in pairs) seem to take the place of the more specialized set of angon + spear that we see in the larger, organized armies of the earlier (late imperial) period. I would suggest this reflects the decentralization and de-professionalization of the army in the early middle ages.

When, precisely, the angon was entirely abandoned isn't clear. They disappear from archaeology after the 7th century, but that's because weapon burial went out of fashion. They continue to appear in art for the next 400 years, but whether that reflects reality, or whether the shape had become 'the way' to draw javelins (and outlived the weapons it was based on) is difficult to say.

What does seem clear is that by the time armies re-specialized (in the later part of the early middle ages), the pilum / angon does not seem - as far as we can tell - to be given the same privileged place it had been in the old Roman model (whose influence had continued well into the early middle ages).