u/Stanchion_Excelsior's different but related answer about Norse mythology and dwarfs here: essentially it seems that perhaps the modern conception of a fantasy dwarf as a stout beardy fellow and the older folklore of dwarfs as supernatural entities are totally different and that people with dwarfism couldn't have been confused for the other.