I've wondered for a while why quillons were not used in the ancient world, or at least were not used very much. What changed about warfare that would cause this design to become popular?
Perhaps the question, rather than "Why were quillons not used in the ancient world?", should instead be "Why were quillons used in the late Medieval period?" Not having long quillons is more common over time and place. For example, the swords of continental and island SE Asia mostly have no guard, or a small guard; Japanese swords a disc guard, or a short stubby cross-guard, or no guard; Chinese swords a disc/cup guard or a compact guard (as on most jian); Indian swords usually have a short stubby guard or a partial basket-hilt; Central Asian and Iranian swords before the 10th century (and often afterwards) have short stubby guards or sometimes no guard; sub-Saharan African swords usually have no guard; and post-Medieval European swords (and swords worldwide influenced by them) usually have complex hilts, simple knucklebows, or partial or full baskets. Long quillons is not some default choice for swords, and their lack needs no explanation.
What are long quillons for then? Fundamentally, we don't know why they were adopted. While we have many martial arts manuals (i.e., fightbooks) from the later Medieval period (see Wiktenauer for many examples and translations), they don't provide this level of detail (we also lack details of footwork, the mechanics of cutting, parrying, and more). As Medieval European martial arts have been reconstructed from these written sources, this kind of detail has been filled in by experimentation and/or "frog DNA" (a Jurassic Park inspired term, meaning to fill in the gaps using other martial arts, e.g., Japanese sword arts). This tells us what can be done with long quillons, but it can't tell us why they were originally adopted or became popular.
What do we know from reconstruction of Medieval European martial arts? We know that long quillons do provide more protection than short simple guards, but far from enough protection to be reliable - if somebody is depending on quillons to passively protect their hands, it is usually fairly easy to hit them in their hands. The late Medieval evolution of more protective guards (e.g., by adding side rings) shows the deficiency in protectiveness of quillons alone. What they are useful for is for manipulating and trapping an opponent's sword. Downward-curved (i.e., curved towards the blade) quillons are more effective for this, at the cost of protectiveness.
What do we know from the sources? The oldest surviving Medieval European fightbook, I.33, covering sword and buckler:
shows the buckler being diligently used to protect the sword hand. Clearly, quillons alone are insufficient. Other than that, it provides no clue to the function of quillons. Where we see the quillons in use in the fightbooks, it is usually offensive use. Perhaps the most primal of these offensive techniques is the "Mordshlag" or "Mordhau" ("murder blow" or "murder strike"; also known by other names), holding the sword by the blade and striking with the quillons:
This example is from the mid-15th century Königsegg manuscript by Hans Talhoffer (and shows a defence against this technique). Talhoffer's fightbooks are predominantly pictorial, with terse text describing the action. In two Talhoffer manuscripts:
we can see more examples of offensive use. For example, following a strike with the quillons - the above - the quillons can be used to pull/wrench the opponent's sword:
Similarly, they can be used to hook/pull/wrench the opponent's neck:
or leg:
The quillons can also be used to provide a stronger grip when thrusting into a gap in the armour:
These are techniques for fighting in armour. For unarmoured fighting, we can also see the quillons used for manipulating the opponent's blade. From Lecküchner's 15th century fightbook,
for unarmoured messer, we have
and the relevant part of the text can be translated as
If he is too strong in the displacement, so that you cannot come to these elements, work with the crosspiece or with the "guarding nail" and push his messer with it aside
(translation from Wiktenauer). Note that the "nail" or Nagel is like a 3rd quillon, at right angles to the crossguard, on the right-hand side of the blade (for a right-handed messer/sword). Finally, from Paulus Kal's 15th century fightbook,
we have a straightforward instruction to hit the hand behind the quillons:
Long quillons appear to provide more offensive advantage over earlier shorter guards. They also provide a small defensive advantage. Thus, they appear to be better. In that case, why were they not used more widely? First, offensive use exposes the hands, and armour to protect the hands is useful (note that I.33, for unarmoured sword and buckler fighting protects the hands with the buckler, and doesn't make offensive use of the quillons). The dominance of long quillons in Europe coincides with the time when hand armour (mail mittens and plate gauntlets) was most common. The use of mail mittens might have made offensive use of quillons more attractive, and the disappearance of plate gauntlets drove a move towards complex hilts and basket hilts.
Second, quillons come with some disadvantages. In close fighting (typically in armour, but possibly without), they can be grabbed by the opponent. They can also catch on clothing or shields, either the wielder's or their opponent's. This is more likely to happen with slicing draw cuts, which is a style of cut widely used in SE Asia, Japan, China, and India (as noted at the start, places where swords without long quillons are usual). There is overlap between slicing draw cuts and long quillons (e.g., in Iran and western Asia), so this would only be a minor disadvantage. As already noted, we don't know details about cutting mechanics in Medieval Europe, so we don't know whether this was a factor.
Thus, long quillons provide offensive benefits, especially when used with hand armour. Otherwise, they provide little defensive advantage compared to shorter simple guards, and are less protective than complex guards, basket hilts, and possibly disc/cup guards (which provide more lateral protection). They bring some minor disadvantages compared to shorter guards (and disc/cup guards).
Finally, two further points:
The very long quillons:
seen on many large two-handed swords appear to be intended for fighting against spears and other polearms. A two-handed spear is a very dangerous weapon to face with just a sword: after the sword is used to parry a spear thrust, the spearman can quickly disengage, in a haf or three-quarter circle around the hilt, and attack again. The very long quillons (and the long grip) mean that the spear has to move much further, and this disengage-and-continue takes much more time. Where the tips of the quillons curve towards the blade, they are more effective for this. (Possibly the side-projections on many two-handed sword blades ("Parierhaken', or "parrying hooks") are also meant to made it harder for a spear/polearm to disengage.)
We also see long quillons as part of complex hilts, where they won't contribute significantly to protection
since the rest of the guard already provides plenty of protection. On rapiers, which due to their length and weight are often rather unwieldy, these will make it harder for an opponent to disengage, similar to two-handed sword vs spear.
For the typical knightly sword and longsword (i.e., hand-and-a-half sword), the weapons are quite agile and the quillons not usually long enough to provide much effect on disengagement time, if used passively. More active and forceful engagement with the opponent's blade is important for providing more control.
