I've always held to the belief that Hannibal's elephants were more valuable for crossing the Alps than they were for actually fighting the Romans.

The Romans had had some experience fighting against elephants. In the Pyrrhic Wars, they first encountered them at Hereclea, where, according to Gary Forsythe in A Critical History of Early Rome, Rome's defeat was attributed partly to its cavalry "being thrown into confusion by the strange sight, smell, and enormous bulk of the elephants."

They faced elephants again at the Battle of Asculum, and:

After a long time, however, as we are told, they began to be driven back at the point where Pyrrhus himself was pressing hard upon his opponents; but the greatest havoc was wrought by the furious strength of the elephants, since the valour of the Romans was of no avail in fighting them, but they felt that they must yield before them as before an onrushing billow or a crashing earthquake, and not stand their ground only to die in vain, or suffer all that is most grievous without doing any good at all. (*Plutarch 21.7)

As the Romans were known to, they came to adapt. At Asculum, the Romans had gained some temporary success against the elephants by using ox-drawn wagons that creaked loudly, and carried soldiers carrying flaming poles and throwing javelins. Realizing that the key was to frighten the elephants, not kill them, they repeated some of these tactics at Messana during Pyrrhus' Sicilian Campaign:

Down they came from their strong places, and hurling their javelins at the elephants compelled them to wheel about and run back through the ranks of their own men, thus causing disorder and confusion there. This gave the victory to the Romans, and at the same time the advantage also in the struggle for supremacy. For having acquired high courage and power and a reputation for invincibility from their valour in these struggles, they at once got control of Italy, and soon afterwards of Sicily. (*Plutarch 25.5)

The Romans would carry these lessons on to their battles against the Carthaginians. During the First Punic War, at the Battle of Panormus, "the elephants were met by showers of missiles, and maddened by their wounds they stampeded back onto their own forces. The confusion was completed when Metellus launched a sally on Hasdrubal's flank and inflicted a severe defeat on the enemy." (The Cambridge Ancient History, 3rd Edition, Volume 7 Part 2) It should be noted that poor, hilly terrain negated the elephants (and cavalry) at the Battle of Adyss, severely hampering the Carthaginian army, contributing in a defeat. Elephants were successfully used at the Battle of Tunis, ending Regulus' African campaign. However, it should be noted that the success lay in tying up the Roman cavalry at the flanks - the Roman infantry in the center withstood and repelled the elephant attacks. (The First Punic War: A Military History, John Francis Lazenby).

What we see here is that elephants have been powerful assets, but that the Romans became increasingly accustomed to and more effective at repelling them. That begs the question - why did Hannibal bring them along?

We should look at Hannibal's use of elephants against the Romans, and I will point to two battles. The first was the Battle of the Trebia River; essentially, Hannibal goaded the Romans into crossing the river in the early hours of the morning without food, and when half of the army was across, the Carthaginians, who had formed a semi-circle around the ford, attacked. Hannibal's elephants successfully drove off the Roman cavalry, but when they turned back against the Roman infantry, they were panicked by javelins, and:

The beasts were panic-stricken, and roused to fury against their own side, and so Hannibal ordered them to be driven from the centre of the fight against the Gallic auxiliaries on the far left wing. There they immediately occasioned what was undoubtedly a rout, and this inspired fresh panic in the Romans when they saw their auxiliaries driven off in defeat. (Livy XXI.56)

At the Battle of Canusium, the elephants again ran amok, causing more damage to the Carthaginians than they did the Romans:

After a long and undecided struggle [Hannibal] ordered the elephants to be brought up into the fighting line, in the hope that they would create confusion and panic among the enemy. At first they threw the front ranks into disorder, trampling some underfoot and scattering those round in wild alarm. One flank was thus exposed, and the rout would have spread much farther had not C. Decimius Flavus, one of the military tribunes, snatched the standard of the foremost maniple of hastati and called on them to follow him. He took them to where the animals trotting close to one another were creating the greatest tumult, and told his men to hurl their javelins at them. Owing to the short distance and the huge mark presented by the beasts, crowded as they were together, every missile went home. They were not all hit, but those in whose flanks the javelins were sticking turned the uninjured ones to flight, for these animals cannot be depended upon. Not only the men who first attacked them, but every soldier within reach hurled his javelin at them as they galloped back into the Carthaginian ranks, where they caused much more destruction than they had caused amongst the enemy. (Livy XXVII.14)

So it looks like Hannibal's elephants remained consistently useful at countering Roman cavalry, but virtually useless against the infantry. To your original question: how militarily significant were they? Well, against, the Romans, I would say insignificant. No elephants were present at Cannae, yet Hannibal still shattered the Roman cavalry and achieved one of the greatest victories of all time.

The elephants played an important role that is sometimes overlooked, though - they helped him get over the Alps. According to Livy: "Where the track was narrow and precipitous, [the elephants] slowed things up considerably; but wherever they were placed in the column, they offered useful protection to the troops, because the natives had never come across them before and were frightened to go near them." (XXI.35.2) As Hannibal was several times betrayed by his native guides and ambushed by the Allobroges, I think that the psychological effect of the elephants' presence was an important advantage that Hannibal had while making his crossing.