Was it because the older Romans clinged to tradition or so that they had atleast some sort of spear unit against cavalry?
The notion of needing a "spear unit against cavalry" comes from video games with rock-paper-scissor mechanics and has no bearing on actual military history. Spears were by far the most common weapon used in antiquity, whether by infantry or by cavalry, and spear-armed infantry would almost always be fighting other infantry.
Indeed, to some extent the entire idea of "units" is a war-game or video-game idea. Professional Hellenistic and (less professional) Roman armies had distinct types of infantry with specialist roles, but tribal forces and the like were organised in battle-lines with little differentiation, and even the Greek soldiers would typically be formed in a single solid phalanx. They would still have some sort of subdivisions, but these would not usually be armed with distinct types of weaponry ("units of spearmen") but instead be things like the retinues of individual leaders or contingents from certain tribes or regions. In most armies, particularly tribal ones, these would not be capable of independent action and manoeuvre. It was one of the demonstrations of the Roman army's skill that they could and did occasionally send cohorts to outflank enemies, launch rear attacks, or react to breakthroughs and enemy movements. But even for them this was a risky kind of ploy, and even for them it would be impossible to use any unit of infantry specifically against cavalry unless you managed to surprise them like Caesar did at Pharsalus. It should also be noted that the examples I can think of tend to involve the more professional army of the late Republic, and not the one of the middle Republic that had the spear-armed triarii.
Before I go too far the other direction, though: spears themselves certainly were useful when fighting cavalry, because of their longer reach. We do have various instances, such as the battle of Pharsalus or Arrian's array against the Alans, where infantry are specifically instructed to use spears against cavalry. (Or their pila, in the case of Pharsalus.) It's just that there usually is no way in a battle of making sure any "unit" with a certain kind of weapon ends up fighting the enemy cavalry, if there even is such a thing as a unit at all. In the examples we have, the general knows they're going to face a cavalry-heavy enemy and organises his army accordingly. He isn't keeping a reserve of spearmen like the triarii.
Nor was it a case of "old Romans clinging to tradition" since the triarii kept their spears in generation after generation, and these same men would in their youth have used the javelin and short sword just like the other soldiers in the legion. (Indeed, they did still carry swords in addition to their spears.) It was no more or less traditional to carry the heavy spear than to carry the javelin.
So why did the triarii carry spears? Or indeed, the rest of the hoplite panoply? Our sources aren't very explicit in explaining why. We do know the tactical role of the triarii, though: they were the ultimate reserve, and would often cover the retreat of the rest of the army. This is probably why their equipment is different. The first two lines of heavy infantry had to attack the enemy, and so carried throwing spears to soften them up and short swords to launch aggressive close-quarter attacks. The more heavily equipped triarii would typically rest on one knee, or even sit down, and watch the battle progress. Usually, they would do nothing much at all. ("Come now, everyone sit on the periphery, just like the triarii," Plautus jokes in one of his plays.)
But if the battle went poorly for the Romans, and they had to retreat, the triarii were there to fall back on. These older, steadier men, who wouldn't panic as easily as the younger soldiers, would take up their heavy shields, form a dense phalanx, and either fight with their longer spears or plant them in the ground to make a deadly fence of spearpoints and fight with their swords, thereby presenting a very formidable obstacle to the tired enemy who thought they had already won the battle. (Indeed, on one occasion the heavy spears of the triarii were given to the first line of hastati to help them block a wild Gallic (infantry) charge.) Since the vast, vast majority of casualties in ancient battles occurred when one army broke and ran and was pursued for miles by the vengeful enemy, this practice seems very sensible in safeguarding against devastating loss. It is also plausible that the younger soldiers were much encouraged by knowing they had the triarii waiting at their back in case things went pear-shaped, and therefore would be much more confident in their own attacks.
Sources: