Well, Ernst Jentsch published Zur Psychologie des Unheimlichen in 1906 and Sigmund Freud's Das Unheimliche is 1919, and both of them turn to simulacra of human beings as components of their respective understandings of "the uncanny." So while Uncanny Valley itself is a recent coinage, we can rule out the general concept as a modern-day invention. (Whatever its proven or unproven actual psychological underpinnings).
However, neither Jentsch nor Freud is talking specifically about dolls in the sense of Bratz or Cabbage Patch Kids. In fact, Jentsch specifically asserts that "A doll which closes and opens its eyes by itself, or a small automatic toy, will cause no notable sensation of [the uncanny]." (trans. Roy Sellars; PDF). Instead, what he (and to a lesser extent Freud, who simply stipulates Jentsch's point here) circles as the uncanny in terms of mock-humans is automata--that is, "the life-size machines that perform complicated tasks, blow trumpets, dance and so forth."
For Jentsch, the uncanny lies in the question of whether something/someone is human or a machine. Critically, this can go in either direction--he identifies the jerking, unconscious movements of a person having a tonic-clonic (grand mal) seizure as triggering the sensation of the uncanny in the same way. But his exemplar par excellence, also picked up by Freud, is E.T.A. Hoffmann's short story Der Sandmann.
From the beginning, in the protagonist's recounting of his childhood nightmare/fever dream/vision/reality, the story blurs the line between human and doll-automaton. Creeptastic Coppelius, the reified "Sandman" of Nathanael's childhood terror, seizes the boy in his father's room and treats him like an automaton:
"We will examine the mechanism (Mechanismus) of his hands and feet."
And then he seized me so roughly that my joints cracked, and screwed off my hands and feet, afterwards putting them back again, one after the other. "There's something wrong here," he mumbled. "But now it's as good as ever."
Fast-forwarding in time, Nathanael is engaged to a (genuinely lovely) woman named Clara. However, through the lens of a telescope, he becomes obsessed with another woman named Olimpia--the lens of the telescope being, in a way, artificial/mechanical eyes. Of course the drama of the story is that beautiful, dancing Olimpia, who always responds, "Ah, ah!" is actually an automaton.
And in the story's finale, when Nathanael and Clara have been reconciled, they go sightseeing one day, observing the land around them from a tower. Clara spots an apparently triffid, and Nathanael wants to take a closer look:
Nathaniel mechanically put his hand into his breast pocket--he found Coppola's telescope, and pointed it to one side.
But before he sees the seemingly-walking shrub, he spots Clara. And his mind flashes back to dancing Olimpia, always through the telescope lens:
He shrieked out in a piercing tone, "Spin round, wooden doll! Spin round!" Then seizing Clara with immense force, he tried to hurl her down, but with the desperate strength of one battling against death she clutched the railings.
Clara survives! But in his madness over the confusion of living and doll, Nathanael hurtles himself off the steeple. Throughout the story, then, the boundary between human and machine is repeatedly blurred, right into insanity and death.
(Freud thinks the real issue in Hoffmann's story is that eyes are a penis and women don't real, but...you know, Freud.)
Scholars of Gothic literature in particular have traced this theme of automata as the creepy uncanny, in an age where they were meant first of all to be a wondrous scientific and mechanical achievement, back to the Romantic reaction to the Enlightenment. The idea of the human body as mechanism or machine (an automaton animated by the soul, if you want to be Cartesian about it) runs strongly through the writing of philosophers like La Mettrie. As Hoffmann's story shows, the idea could be inspiring to some, fearsome to others, and most importantly, both at once.
The theme of human/machine as horror is so prominent in 18th and 19th century Romantic and Gothic literature that Terry Castle even called her anthology of "greatest hits" academic articles The Female Thermometer and the Eighteenth-Century Invention of the Uncanny.
...Of course, Castle is not a medievalist. And just as the medieval Near East, Byzantium, and (somewhat later) western Europe had both literary and practical traditions of awe-inspiring automata, the era's literature reveals a sense of the uncanny surrounding them.
However, as (AskHistorians AMA alumna!) E.R. Truitt points out, the creepy sensation for medieval authors wasn't a question of human and/versus machine. Instead, it was the twilight between life and death. Automata guard tombs, including ones meant to be a direct replication of the deceased; they guard castles populated by spectres. In some stories, automata resembling dead people are employed as a purposeful fraud to pretend the person is still alive.
So the overall point is: what is uncanny to the people of one era might not be to those of another, or it might be equally uncanny but for different reasons.
As to why some people today find antique dolls creepy, I was unable to find any solid research beyond the usual "dolls in horror movies" (fun fact: in the 19th century, it's more often the puppet masters who are the bad guys) as well as some info that suggests the phenomenon isn't limited to antique dolls. But this will be quite interesting in light of the above, I think:
In Francis McAndrew and Sara Koehnke's study "On the Nature of Creepiness," (PDF) their respondents rated "collecting something" as the creepiest hobby--provided that something was "dolls, insects, reptiles, or body parts such as teeth, bones, or fingernails." McAndrew and Koehnke identify, overall, creepiness as the sensation of ambiguity over whether something is or isn't a threat. The things that people find creepy to collect are, they point out, either things that humans have an instinctive fear of--snakes and spiders--or things that relate to the living/dead boundary--pulled teeth, clipped fingernails, or outright bones.
Or dolls.
Our sense of the uncanny, it seems, has more traces of the medieval than one might expect from this side of industrialization and the microchip.
Dolls, and other similar simulacra (statues, idols, icons) are common to all cultures in all times. I think the "antique" dolls you are referring to are porcelain (or 'bisque') dolls which were popular in the 19th and early 20th century with a brief resurgence among adults in the 1980's & 90's (cf. Robertson). They are by far the creepiest. Any search on ebay for porcelain dolls bring up kistchy "haunted" dolls in the style of the porcelain doll.
You've alluded to why they are creepy in your question. They are uncanny. They themselves represent small children that are surprisingly life-like: real hair, moving eyelids, glass eyes that stare straight ahead and, though apparently children, they have a certain seriousness to their face that is normally associated with adults. Not to mention their "skin" is porcelain, whose color and visible texture more closely resemble human skin then, say, the fabric, other types of clay, or wood that was used to make most dolls before this period.
This specific type of doll was not initially produced for children. They were produced at first for adults as models for women's clothes. The more realistic the doll, the more they were sought after (Robertson 22). Obviously, this was a practical concern, because a realistic doll could adequately, albeit in miniature, provide a more clear picture of how the finished design would appear. It's safe to assume then, that the hyper-realistic appearance of the dolls made for children stems from their original purpose as models and the practical demand for realism.
This popularity coincided with children's changing social role in the western world. The emphasis on 'childhood' as distinct from 'adulthood' became more pronounced and there was more of an effort to get them into schools and, this being the dawn of the age of mass-production, markets sprung up in which clothes, books and toys were made specifically for children (Robertson 25). The boom in demand for these dolls was from the mid 19th century until around the 1970s, after which they were primarily sold on the antique market (Robertson 28-29).
Their realism, produced for other, practical, purposes, is definitely the source of the uncanny feeling. However, while something life-like often fills us with wonder, it just as often fills us with dread. Their gaze is focused, stuck and wide-eyed and although they don't actually move on their own, they appear that they can or are about to move due to their realistic appearance. That this is all very unsettling is implied by your posing this question and I think we can safely say that writings such as Hoffmann's Der Sandmann, or Rilke's "Dolls: On the wax dolls of Lotte Pritzel" in which he describes the concept of the doll as a "painted watery corpse" (Paris Review), and the uncountable other sources (without failing to mention Child's Play) confirms that dolls and other life-imitating objects have always occupied a place of terror and awe in western culture.
So, perhaps it is not that people didn't have a sense of the uncanny valley in the past, but rather that children specifically do not have that same sense.
Research has shown that children do not experience, or experience differently, the uncanny valley phenomenon. Psychologists Henry Wellman and Kimberly Brink recently studied how children respond to robots and found that "young children (children younger than 9 years), unlike adults, don't think robots, even those that look very much like humans, are very creepy at all" (NPR). This conclusion comes from the observation that children tended to not be 'creeped out' by human-seeming robot but, contrastingly, adults found human-like robots unsettling. I think we can agree that this has a connection to how children interact with dolls as well. Given this, children might actually prefer life-like dolls, while adults may find them unsettling.
If we agree with the conclusion that children are more comfortable attributing human characteristics to objects, then it is children who can be said to be a bit uncanny to us as well. How many horror movies use children as the central ghoul? Childhood experience is felt and represented in adult culture as something lost and foreign, not to mention traumatic and confusing which is - as Freudians would say - repressed. Thus, childhood is itself uncanny (at once familiar and foreign), and the perceived creepiness of this specific type of doll may be the return of that repressed otherworld of childhood in which we were comfortable with attributing mental faculties to life-like objects.
As for the people who collect them as adults, that would perhaps be a different discussion.
Sources:
https://www.taylorfrancis.com/books/9780415944519
https://www.theparisreview.org/blog/2018/05/25/the-unfortunate-fate-of-childhood-dolls/
Note: Not an expert; just had some time for research.
Edit: grammar, elaboration. Added citation.
Hi, everybody! So, you've probably clicked on this thread because it has so many upvotes and you assume that means there's an answer - but that's not how /r/AskHistorians works. In general, people upvote threads here because they want an answer. The thing is, it can often take time for a good answer to be written. Our mission is to provide users with in-depth and comprehensive responses, and our rules are intended to facilitate that purpose. That's why we remove comments that are very short, based on speculation, or just plain inaccurate. Making comments asking/complaining about the removed comments simply makes the problem worse. So please, before you try your hand at posting, check out the rules: we don't want to have to warn you again or give you a temporary ban for ignoring them.
While you wait, you can check out places our already-answered questions are featured, including Twitter, the Sunday Digest, the Monthly "Best Of" feature, and now, Facebook. It's very rare that a decent answer doesn't result in due time, so please do come back to this thread in a few hours. If you think you might forget, send a Private Message to the Remind-Me bot, and it will ensure you don't!
Finally, while we always appreciate feedback, we ask that any comments about our moderation style and rules be sent to modmail, or a META thread. Please be aware that we've considered and discarded the idea of an "answered" flair and an automatic "free talk" comment thread many times already. Thank you!