You see them so often in video games but they seem really ineffective compared to swords and axes.
I would begin by saying that video games are generally atrocious at accurate weapon and armor design, and that some of the fantasy type war hammers with massive heads would only have existed as tools, if they existed at all. A weapon needs to be able to be wielded with agility to be effective in combat.
War hammers were small weapons, meant to be easily manageable in one hand. Ewart Oakeshott's A Knight and His Weapons describes the typical war hammer as being about two to two and a half feet in length. Hammer heads weighed around three pounds and consisted of a striking head (that often contained many gripping teeth), and a piercing spike.
War hammers are described as being carried secondarily to a sword and/or lance and were primarily used in close range armored combat - a major disadvantage of the weapon being its short reach. The major advantage is the ability to transfer energy through armor, as well as to cause disabling dents. Armor was very carefully fitted and warped plates could easily reduce range of motion. It is important to note that swords are also capable of biting into plate and performing this same function. War hammers were just more specialized for the task.
While the weapons popularity is closely correlated with the usage of complex plate armor of the later medieval period (it was notably popular during the Hundred Years War), there are images of warriors armed with war hammers from the 13th century as well. I can't find an image online but Oakeshott mentions a tomb effigy in Malvern Abbey that features an unnamed warrior in 13th century armor armed only with a small hammer and a bucker (a sort of small round shield, usually 9in-12in across).
Sources: Ewart Oakeshott: The Archaeology of Weapons, A Knight and His Weapons, A Knight and His Armor
Mike Loades: Swords and Swordsman
I would additionally recommend watching videos by Scholagladiatoria on Youtube related to the topic.
Unfortunately there no surviving manual that covers war hammer usage (to my knowledge) but people like Matt Easton attempt to speculate on usage by examining historical accounts (particularly artwork) and applying knowledge of other, better documented, weapons of the time.
Edit: I should clarify that I am writing this under the assumption that the war hammer is a small weapon as described by Oakeshott (who is famous for his sword typologies, but doesn't dwell very much on war hammers). I have seen people referring to a war hammer as a longer polearm with a hammer head, which Oakeshott would describe as a polehammer.
Hi there! Here's a response I gave to this question about a year ago. Super good and interesting question.