From what I can find, it was originally pronounced with the "h" sound in Roman times, but I cannot find any information regarding when the change happened.
(Asking to verify that the movie "The Little Hours", about a group of nuns in the 14th Century, is also a subtle joke)
Thank you!
The word seems to have entered English from Old French in the thirteenth century without the h, as we now pronounce it. A search of the Linguistic Atlas of Early Middle English for "hour" returns eleven localised manuscripts from the thirteenth and early fourteenth century, ten of which predominantly or exclusively give forms without the h (ure/oure/our etc). This is backed up by the earliest citations in the MED entry. These spellings almost certainly reflect a pronunciation in which h was not sounded. An Old French specialist could probably provide a better sense of when exactly the initial sound dropped off the Latin word (we still use it in direct Latin loanwords like modern English "horology").
The written h starts to be commonly attached to the word again in the fourteenth and fifteenth centuries (although not always), likely due to the influence of French/Latin orthography (and perhaps creeping standardisation?) on English writers and copyists. Although it is always dangerous to generalise about local dialect, the written change does not usually seem to have been accompanied by a change in pronunciation. In the B-text of William Langland's alliterative poem Piers Plowman (c. 1370-80), for example, "houres" alliterates with other initial-o words: "overhuppen", "other" and "offices" (Passus 15 line 402, in the text edited at the Piers Plowman Electronic Archive - all ten of the manuscripts transcribed for this edition give initial h).
The vowel is a bit trickier, as these are more prone to variation based on period and local dialect. For convenience's sake, in the dialect we tend to think of as "Middle English" (ie that of the Midlands and London c.1400), ou in "hour" was probably pronounced like IPA [u:], as in Modern Standard English "boot". During the so-called "Great Vowel Shift" of the late Middle Ages and Early Modern period, the sound migrated toward our modern usage. In some dialects it developed the slight diphthong that distinguishes it from more straightforward words like "house" - a better linguist than I will have to explain why that was the case.