There is a mute and an aspirated h in French. The latter is not pronounced either but it's not elided, too (and is shown in IPA transciption as an ʔ). It is mostly found in words of Frankish origin (la hachette = the hachet, le hagard = the haggard, la herse = the harrow) and in some later borrowings from the "exotic" languages like Arabic (le hadith). The former is found in words inherited from Vulgar Latin (l'heure, l'herbe, l'hiver) and in the borrowings from "boring" languages like English.

That puts the first loss sometime before France fell to Franks and a second loss sometime after their language fused with that of their French subjects.