The notion of semi-human monsters or peoples is hardly unique in ancient cultures. Just look at the prevalence of the idea of dog-headed peoples, or cynocephaly, going back to at least Egyptian mythology and as we will see momentarily, often associated with the Blemmyes. The particular idea of headless people, with eyes and so on on their chest, living on the extremities of Africa goes back to Herodotus, or at least he is our earliest source. Near the end of his discussion of peoples of Libya (Africa), he notes that:

For the eastern side of Libya, where the wanderers dwell, is low and sandy, as far as the river Triton; but westward of that the land of the husbandmen is very hilly, and abounds with forests and wild beasts. For this is the tract in which the huge serpents are found, and the lions, the elephants, the bears, the aspicks, and the horned asses. Here too are the dog-faced creatures, and the creatures without heads, whom the Libyans declare to have their eyes in their breasts; and also the wild men, and wild women, and many other far less fabulous beasts.

Herodotus' offhanded inclusion of these fabulous peoples is not particularly surprising as the extremities of the world in the North (Hyperborea, the Amazons, etc.), East (India), West (Celts, Atlantas) and South (Ethiopia) had since our earliest writings been viewed by the Greeks as semi-mythical regions with usual characteristics (India, for example, was supposed to have two of every season in one year), as well as strange animals and peoples. Indeed, Homer's Odessey essentially begins with Poseidon taking a trip to Ethiopia:

Poseidon had gone among the far-off Ethiopians—the Ethiopians who dwell sundered in twain, the farthermost of men, some where Hyperion sets and some where he rises (1.23-4)

So it should be equally unsurprising that this idea of a headless people in the extremities of Africa permeated ancient geographical literature. On this basis, the idea of strange peoples and animals at the peripheries of the world likewise permeates medieval ideas about the world. As in ancient literature (and National Geographic), medieval readers enjoyed descriptions of the strange and wondrous peoples and animals in the far off regions of the world. For example, in a popular twelfth century encyclopedia, the Imago mundi, Honorius Augustodunensis interrupts his geography of the world after describing India to give an account of "About Monsters" and then "About Animals", beginning the former by noting that:

There are certain monsters there which some consider humans and others consider animals.

Sunt ibi quedam monstra que quidam hominibus quidam ascribitur bestiis.

Of course, a Blemmy-like people makes an appearance. This is actually more like the 'Epiphagi' (seen just below the Blemmy on the Hereford map), who have their eyes on their shoulders rather than chest:

There are others without a head who have eyes on their shoulders, for a nose and mouth they have two holes in their chest and they have hair like animals.

Sunt alii absque capite quibus oculi sunt in humeris pro naso et ore duo foramina in pectore, setas habent ut bestie.

See also the famous depiction in British Library, Harley 2799.

For the Hereford map itself, the relevant lineage begins with Pliny the Eldar's Natural History. He explains that in the extremities of Ethiopia, there are a range of strange peoples, including of course the Blemmyes:

The river Nigris has the same characteristics as the Nile; it produces the calamus, the papyrus, and just the same animals, and it rises at the same seasons of the year. Its source is between the Tarrælian Æthiopians and the Œcalicæ. Magium, the city of the latter people, has been placed by some writers amid the deserts, and, next to them the Atlantes; then the Ægipani, half men, half beasts, the Blemmyæ, the Gamphasantes, the Satyri, and the Himantopodes. (5.8)

He goes on to describe each of these, providing a standard description of the Blemmyes:

The Blemmyæ are said to have no heads, their mouths and eyes being seated in their breasts

Blemmyes traduntur capita abesse, ore et oculis pectore adfixis

From here, in the third century, a guy called Gaius Julius Solinus wrote a book on the marvels of the world, variously called the Polyhistor, De mirabilibus mundi (On the marvels of the world) or Collectanea rerum memorabilium (Collection of remarkable things). It is mostly a book about geography, drawn largely from people like Pliny, but it became very popular in late antiquity and the Middle Ages.

They believe that the Blemmyes are born deprived of the part where the head is, yet have a mouth and eyes in their chest.

Blemyas credunt truncos nasci parte qua caput est, os tamen et oculos habere in pectore. (31.5)

Finally, Isidore includes Solinus's description in his Etymologies (probably the most important encyclopedia in the Middle Ages), and it is this text exactly that can be found in the inscription on the map:

People believe that the Blemmyans in Libya are born as trunks without heads, and having their mouth and eyes in their chest, and that another race is born without necks and having their eyes in their shoulders. (11.3.17)

Blemmyas in Libya credunt truncos sine capite nasci, et os et oculos habere in pectore. Alios sine cervicibus gigni, oculos habentes in humeris.

And indeed, the reason that we find two of these headless peoples on the map is because Isidore adds a second sentence, not found in Solinus, explaining that others have the eyes on their shoulders, rather than their chest. (Nb. the translation I've given here, from Barney et al. is a tad misleading, in that the Latin doesn't strictly state that this is 'another race', although that may be Isidore's intention and, as we can see, that is definitely how some medieval readers interpreted it.