The Turn Of The Screw was first published in serial form, which seems to have been typical for late 19th century literature but is hard for me to wrap my brain around now given the novella's ambiguous and unsettling themes. Occasionally in discussions of the novella you'll run into people asserting that 19th century readers couldn't possibly have grasped elements like the main character as an unreliable narrator, or various forms of psychological subtext, but this seems like it's selling contemporary readers short -- what were contemporary responses to the novel like? Either public responses in a critical context or private responses -- were there objections to the novel's content? Was it seen as a continuation of a preexisting literary tradition in ghost stories, or something novel?
In the first place, serial publication was great for mystery and suspense-driven genres! Wilkie Collins and the other 'sensation novelists' really mastered this form in the 1860s and 1870s, using the breaks between installments to heighten anticipation.
When it comes to whether contemporary readers could understand "The Turn of the Screw," James himself actually thought the story was too easy to understand. He wrote it, he claims in his NY Edition Preface, as a "potboiler," a cheap, fluffy piece of genre fiction that would be on-trend and make money. And it was! There was an enormous appetite for ghost stories and Gothic-tinged fiction in the 1890's -- Bram Stoker, Sheridan LeFanu, and R.L. Stevenson all made major contributions to this genre in the decade, and "The Turn of the Screw" was wildly successful. It received some great reviews: the Standard called it "a work of extraordinary power" and the Athenaeum praised the work because "the author makes triumphant use of his subtlety" (Both quoted in The Academy, Dec. 31, 1898). Other writers loved it -- Oscar Wilde (not always a huge fan of James) wrote admiringly of it, and H.G. Wells even wrote to James to demand a fuller explanation of the governess's character. James's response to Wells is an extremely Jamesian non-answer about "I had to rule out subjective complications of her own -- play of tone etc.; and keep her impersonal save for the most obvious and indispensable note of neatness, firmness and courage" (I don't have this whole letter, but it's quoted in the introduction to the Penguin edition, 17). Virginia Woolf wrote about it at some length, too, in the 1920's or 30's. To suggest that all of these readers were somehow incapable of perceiving the story's Freudian themes or its narratological complexity sounds silly to me. This type of story was extremely popular at the fin-de-siecle for precisely those reasons: because using 'uncanny' or fantastic themes and intricate Gothic narration was a way to plumb the repressed desires and unconscious horrors that lurk beneath the surface of what's socially acceptable.
Sources: "Book Reviews Reviewed," The Academy, 31 December 1898. --This has a nice summary of contemporary reviews of the novella.
Curtis, Anthony. "Introduction" to Henry James, The Aspern Papers and the Turn of the Screw (London: Penguin Classics, 1986).
Beidler, P.G. Ghosts, Demons, and Henry James: The Turn of the Screw at the Turn of the Century (University of Missouri Press, 1989).