I can't answer this for all of rural America, but there's one interesting possibility for communities of Scottish heritage. In Margaret Bennett's Scottish Customs from the Cradle to the Grave, there's a section on fairy changeling lore. Fairies were widely held to steal human babies, replacing them with changelings. There are many, many stories about this in Scottish Gaelic. The accounts Bennett collects strongly suggest that part of the origin of these legends may be real-world experiences with what we might today call children with special needs. One informant recounts seeing a changeling himself, a young man who could neither speak nor move much without help, who was kept in an upstairs bedroom until he passed away at about the age of 20. In changeling stories, the switch is often discovered after the mother notices a change in the child: it cries more, it fails to thrive. The changeling is usually tricked into revealing itself (by, for example, the mother making tea in eggshells, causing the changeling to exclaim, "I've lived a thousand years but I've never seen the like of that!") The real baby can then sometimes be recovered by killing the changeling. If the family kept the changeling, it would normally waste away at a young age.