Oh boy, this entire answer is going to skirt the 20-year rule like nobody's business, but I love the question and I'm ecstatic to be able to talk about this (fun fact about me: I love children's lit. I nearly did my undergrad thesis on Nancy Drew...I ended up doing it on fairy tales instead). To avoid spending the entire time just talking about the past twenty years, I'll instead take you on a journey through the evolution of children's literature as a genre and show you how that led to the creation of "young adult" literature.

Part 1: Definitions and The Beginning

First, a definition: children's literature is (generally) defined as a piece of literature where the implied/intended reader is a child (rather than a story about a child, because stories ABOUT children don't necessarily imply that the story is aimed FOR children's consumption; see: everything Stephen King writes starring children ever). Btw, "implied reader" is also a technical term, generally referring to the fact that texts suggest in their subject and style the characteristics of the readers best equipped to understand and respond to them. Now, we may have different ideas of what defines being a "child," but for the purposes of this answer we're going to go with the good old "a child is anyone between the ages of 0 and 18." You can see where the answer to your question is already getting sticky, because YA is definitionally a subset of children's literature, given that it's aimed generally towards adolescents and "young adults" (generally between the ages of 13 and 21).

At this point, we're just going to make a super quick sidenote: Texts can have dual audiences. Just because the implied reader is a child doesn't mean that an adult can't find meaning or enjoyment in it. Texts can and are written for the enjoyment of multiple audiences. That being said, I think the concept of "the intended audience for [this book] is a child" isn't a particularly hard one to grasp.

Now that we've gotten those little issues out of the way:

1. To talk about YA, you have to talk about children's literature.
2. To talk about children's literature, you have to talk about the Victorians.

I'm going to skirt around the actual intricacies of the Victorian conception of childhood, because frankly that's not my specialty and you'd get much more luck on that front asking a question specifically about that, but the long and short of it is that the modern conception of 'childhood' as a section of life distinctly separate from adulthood (and thus a section of life that required specific tools, literature, and education distinct from adult things) arose in the early Victorian era; with that came the idea that there needed to be literature specifically aimed at children. The idea that children were distinctly separate beings from adults was around before that, but the Victorian era was the first time that "child=different" was truly equated with "children need different reading material from adults." Children’s literature as a genre fundamentally relies on the belief that 'child' is a different category from 'adult,' that they’re somehow different and require different things out of literature.

Sidenote: John Newbery is generally credited with creating the first 'modern' children's book, "A Little Pretty Pocket-book," in 1744, and made enormous early contributions to the field aimed mostly at toddlers and young children learning how to read; other early contributions were heavily moralistic tales like The Governess (1749), the first full-length novel for children. However, 'children's literature' as we think of it did not really ramp up until ~the 1820s/1830s. You probably know the name Newbery primarily from the Newbery Award, which was named after him in recognition of his achievements in children's literature and is the highest prize awarded in children's lit.

Anyway, the rise of this belief led to an outpouring of what we now call 'early children's literature': some of the earliest successes were in revised fairy tales (hey Grimms!), which were edited and heavily sanitized once they began to be marketed towards children rather than adults and scholars. The rise in fairy tale collection and dissemination was fueled largely by the 19th century romantic nationalist movement and scholarly ambition, but scholars (and particularly the Brothers Grimm) found that editing their tales and marketing them towards children paid the bills significantly better than keeping them as scholarly works. Other successes include Swiss Family Robinson (1812) and "The Nutcracker and the Mouse King" (1816)--aka, the story that the ballet is based on.

"Now wait, this is all very interesting, but I asked about YA. What about that?" Don't worry, I'm getting there. We've got to get through the Golden Age first, though, because YA doesn't exist without the Golden Age of Children's Literature.

(continued below)