There are no surviving contemporary sources from Joseph Smith's childhood (with the exception of a school attendance record, I think).

As far as I know, the most direct source for Joseph Smith's childhood is from his mother, Lucy Mack Smith, who wrote a book about her son and the rest of her family after he was killed in 1844. She briefly writes about his childhood--mostly about complications he suffered after contracting typhoid fever that resulted in him needing surgery on his leg--but she largely doesn't say anything about Joseph in her narrative before 1820 (when he was 14 years old). She explicitly noted the omission:

yet I am aware that some of my readers, in this, will be disappointed; for, I am led to suppose, by questions which are frequently asked me, that it is thought by some, I will be likely to tell many very remarkable incidents, connected with his childhood; but, as nothing occurred during his early life, excepting those trivial circumstances which are common to that state of human existence, I pass them in silence. (Lucy Mack Smith, History, 1845, p. 71--published version here.)

Basically, she said he wasn't anything exceptional as a kid. She wrote that as a young teen, "Joseph was a remarkably quiet, well disposed child." (p. 72)

According to Joseph Smith's own history, from 1823 to 1827 (when he was 17-21 years old) and before he received the Book of Mormon, he was given "instruction and intelligence" by a heavenly messenger once each year. Lucy Mack Smith wrote briefly about this particular time period, and the removed comment may have had this passage from her history in mind:

From this time forth, Joseph continued to receive instructions from the Lord; and we, to get the children together every evening for the purpose of listening while he imparted the same to the family. I presume we presented an aspect as singular, as any family that ever lived upon the Earth face of the Earth: all seated in a circle, father, mother, sons, and daughters, and giving the most profound attention to a boy, eighteen years of age, who had never read the Bible through in his life; for he was much less inclined to the perusals of books then any of the rest of our children, but far more given to meditation and deep study. . . . During our evening conversations, Joseph would occasionally give us some of the most amusing recitals that could be imagined: he would describe the ancient inhabitants of this continent; their dress, mode of travelling, and the animals upon which they rode; their cities, and their buildings, with every particular; he would describe their mode of warfare, as also their religious worship. This he would do with as much ease, seemingly, as if he had spent his whole life with them. (pp. 86-87)

*Edit: I should probably add that Lucy Mack Smith's history is considered somewhat problematic by some historians. But despite all of its messiness, her narrative is frequently cited in biographies of Joseph Smith because it is the only detailed source about his childhood and early youth. It has the problems common to nearly all reminiscent accounts--some of her dates/chronology are off, she sometimes conflates events together, etc. And it is an account of an affectionate mother writing about her dead sons who she believed were literal martyrs, so of course her narrative would be colored by that. Some also argue that she was motivated to write her book in order to reestablish the religious authority of her surviving family members--namely, Joseph's little brother William who was frequently at odds with Brigham Young--and so she reframed events in her narrative to reflect that.

So, to me, anyway, knowing that raises some unanswerable questions about that last passage I quoted above--did Joseph Smith really tell "amusing recitals" to his brothers and sisters about the people in the Book of Mormon? (I'm not aware of any other times where he was recorded as telling stories with similarly detailed extra-scriptural content.) Or was this a subtle reframing of events by his mother to put his family members right in the middle of the LDS Church's origins? By this telling, his immediate family are his very first converts and privy to knowledge that no other member of the church knew. Interesting stuff, I think.