Obviously nobody was around writing stuff during prehistory, so this image of a caveman had to come from someone's imagination. When, where, and why did this come to be the image of a caveman? My first thought was the Flintstones, but it seems like they relied on some familiarity with the imagery. Is this idea from colonialism and stereotypes of "primitive" natives?
French paleontologist, Marcellin Boule, was the first scientist to describe the Homo neanderthalensis as ape-like in the 1920s, probably leading to the ooga booga part of your question (also, an 1886 short story by Andrew Lang first described the practice of clubbing a woman over the head and dragging her home; cavemen behaviour was seen as primitive, ‘not like us‘). This version of the Homo became to be used interchangeably with the term caveman (although science tells us there were dozens of species of the genus Homo before Homo sapiens monopolized being a human; e.g. the Homo denisova was only discovered in 2010). This term is derived from the fact that the earliest traces of ancient Homini were found in caves, most famously paintings, long before archaeologists unearthed more substantial evidence of their culture. Considering your clothing question: At first, I thought Hollywood, as they have shaped many historically inaccurate looks collectively accepted to be the real thing (e.g. in the monumental antiquity movies popular in the 1940s and 50s). But then I found illustrations of pelt-wearing hunter-gatherers dating back to the 19th century (e.g. the cover of Pierre Boitard‘s 1861 novel, Paris Before Man), so I‘d assume the sentiment of wearing fur is rooted simply in the image of cavemen hunting animals, and that, of course, they would fashion clothing out of their skins (as we still do today). The cliché one-shoulder-strap fur toga seems unlikely though because if you lived somewhere warm enough to not cover your shoulders and legs, fur probably would have been to hot for clothing. Characteristics like that were popularized in comics beginning in the 1910s; in those iterations cavemen were often depicted as living alongside dinosaurs (which they actually have missed by an unfortunate 60 million years).
Conclusion: The image of the caveman was first shaped in the 19th century following the discovery of ancient human species (Homo neanderthalensis was found in Germany in 1856), depicting him as a primitive version of us. This sentiment was build on in the early 20th century through comics and movies.
Other commentators have hit the nail on the head pretty well. There's a lot of intermixing factors for where this conception came from
The general belief in the "primitive-ness" or "apeness" of the Neanderthals, due to them being the first pre-modern human fossils ever found. The first Neanderthal skulls were found even before Darwin' published Origin of Species, so you have to remember that a lot these early interpretations were based upon the lack of the fossil record, how humans evolved (or if they evolved at all), and an over emphasis on "primitive" traits seen in the Neanderthal and H. heidelbergensis fossils. Many of these early fossils were found in caves, which drove the "caveman" conception.
Eurocentrism played a big role in shaping these conceptions as well. I mentioned in an early comment about the racism in construction our evolutionary tree. Early paleoanthropologists often either believed in unilineal origins (one single line of descent) or polygenism (multiple origins, not to be confused with multiregionalism). The unilinealist perspective would place all modern humans in order of evolution, with White Europeans and the top, and Black Africans at the bottom, and then just below Africans would be Neanderthals and Homo erectus ("Java man" was discovered in 1898). The multiple origins perspective was similar, but all human species evolved independently in different areas. This was still used to argue for a racial hierarchy of evolution, stating that peoples like the Khoi san were evolutionary relics. The racist and colonialist attitudes towards hunter-gatherers in Africa shaped the conception of Neanderthals and other "primitive cavemen", since in their mind, they were almost the same thing. This also had a huge effect on interpreting the fossil and archaeological record in Africa and Asia.
The persistent need to separate ourselves (modern H. sapiens) as unique in our cultural abilities, language, art, etc. This involves both of the above historically, but it still persists in arguments to this day about what defines us as a species (beyond morphology). Conceptualizing earlier species as "less human" serves this to define our uniqueness.
I recommend (as I did below) checking out:
Debating humankind's place in nature, 1860-2000 : the nature of paleoanthropology by Richard Delisle. (2007)
For a more recent perspective on how Eurocentrism and Colonialism have shaped out conceptions of the emergence of modern humans, Arthreya and Ackermann have a great chapter in a book about the subject. Here's the pre-print accessible version:
Athreya, Sheela, and Rebecca R. Ackermann. 2018. “Colonialism and Narratives of Human Origins in Asia and Africa.” AfricArXiv. August 18. doi:10.31730/osf.io/jtkn2.