Although our stereotype of prehistoric humans dates from the 19th century, much of the stereotypical "caveman" imagery was borrowed from a couple of older sources:
- According to ancient Greek and Roman geographies, the Trogodytai (Τρωγοδύται) were an indigenous people of North Africa and the Red Sea region. Account differed in the details, but most ancient writers described the Trogodytai as simple hunter-gatherers who lived in the wilderness, subsisted on small game, and spoke in grunts, hisses, and squeaks. Many manuscripts include an "l" in the name (Troglodytai), which changes the meaning to "cave-goer" (τρώγλη, "hole, cave" + δύτης, "one who dives, descends, enters"). This variant, and the association with cave-dwelling, would win out in the long run; it also gives us the modern word "troglodyte" and the scientific nomenclature for chimpanzees (Pan troglodytes). If you take a look at late medieval illustrations of the Troglodytai, you'll see caves, clubs, animal hides, and bushy hair, motifs that would go on to inform later depictions of prehistoric humans. For example, take a look at the Troglodytae in Le secret de l'histoire naturelle contenant les merveilles et choses mémorables du monde (15th century): BnF, MS Français 22971, fol. 13r.
- The medieval "wild man" (homo silvestris) was a semi-mythical character that appeared frequently in late medieval literature and art, especially in German-speaking regions. Wild people were usually depicted with bushy body hair (men and women), carrying clubs, and living a simple hunter's life in the forest. Perusing late medieval and early modern depictions of wild people, you'll again find lots of "caveman" imagery: e.g. Albrecht Dürer's "Sylvan Men" (1499); Martin Shongauer, "Shield with Greyhound Held by Wild Man" (1470-91); an anonymous Basel wall tapestry (1470/80); and the illustrations of Sigenot in Universitätsbibliothek Heidelberg, Cod. Pal. germ. 67, fols. 15v--22r.
It's worth noting that neither the Troglodytai nor the homines silvestres were considered "prehistoric" per se. Both streams would go on to influence European depictions of real-world peoples, such as indigenous Canary Islanders, Americans, Africans, and Australians. The same stock imagery was recycled again with the discovery of prehistoric hominids who had lived in the caves and forests of Europe.
Sources:
Bärbel Auffermann and Gerd-Christian Weniger, "The Neanderthal – Culture Bearer or Savage Man? A Brief Review on 150 Years of Perception History," in Roots - Wurzeln der Menschheit. Katalog zur Ausstellung im Rheinischen Landesmuseum Bonn, ed. Gabriele Uelsberg (2006), 141-196.
Richard Bernheimer, Wild Men in the Middle Ages: A Study in Art, Sentiment, and Demonology (Berlin: De Gruyter, 2013; original 1952).
Timothy Husband, The Wild Man: Medieval Myth and Symbolism (New York: MET Publications, 1980).
Stefan Bradt, Die Bedeutung der Wilden Leute im Mittelalter (Munich: GRIN Verlag, 2001).
Robert G. Morkot, "Trogodytae," Oxford Classical Dictionary (1949).
Leonie von Wilckens, "Das Mittelalter und die 'Wilden Leute'," Münchner Jahrbuch der bildenden Kunst 45 (1994): 65–82.