I'll preface this by saying that this is more a literary question than a historical one; you might get better responses in /r/AskLiteraryStudies.

In A Short History of Fantasy, Farah Mendlesohn and Edward James write the following, in the context of a chapter on the 1970s and, in particular, Stephen King:

Horror fiction had a separate existence from the fantastic at least from the eighteenth century: even when the cause of horror is supernatural, horror fiction is mostly staged in a world we recognize, and hence has an appeal to readers for who other readers have no interest. Horror fiction is also peculiarly cyclical. Its popularity rises and falls as new taboos are identified, and are in their turn exhausted. ^1

However, Mendlesohn and James are unclear on when exactly in the eighteenth century these roots actually are; they do talk about horror through its relationship to Gothic fiction, which arose in the 1760s and lasted in its first iteration through the 1820s (16), but Gothic is much more interested in landscapes and meditations on the sublime, and their revelations of the answer to the mystery tend towards the anticlimactic, which detracts from the cathartic purposes of horror. Also somewhat contradictorily, elsewhere in the book Mendlesohn and James claim that horror "emerged as a separate genre" in the twentieth century before merging with fantasy again in the late twentieth century (22). In other words, it seems, the roots of horror in terms of tropes and conventions started to pop up in the eighteenth century in Gothic fiction and, later, ghost stories, but those conventions and intertextual links, reader expectations etc. didn't seem to coalesce into a 'genre' until much later.

Briefly, too, Roz Kaveny defines the "central aim" of horror (in a passage where she contrasts it to dark fantasy) as "creat[ing] catharsis by confronting the reader with a world in which the worst thing happens to the characters with whom we identify."^2 In horror, the characters are destroyed, bodily and/or mentally/spiritually, by this 'worst thing,' whereas in dark fantasy the protagonists tend to suffer but survive, perhaps mutilated, but wiser. If we take this as a definition, the Gothic starts to look even less like horror, as the supernatural threat is typically explained away though rationality, or defeated by other means.

However, the word 'catharsis' may give you another avenue to follow. IIRC, Aristotle talked about catharsis in his description of tragic theatre as the highest form of art (?). He viewed the purpose of theatre with such heightened emotions as allowing the audience to build up and then release stores of negative feelings, which he argued was beneficial. My recollections on this score are sketchy, though - perhaps someone else can provide a more concrete description and a specific source?

^1. Mendlesohn, Farah and Edward James. A Short History of Fantasy. Middlesex: Middlesex University Press. 2009. 112.

^2. Kaveny, Roz. "Dark fantasy and paranormal romance." The Cambridge Companion to Fantasy Literature. Eds. Edward James and Farah Mendlesohn. 2012. 217.

Edit: some words. Also, the sources I've cited deal almost exclusively with English-language material; /u/coree's answer is fascinating to me because I've never explored other European literatures.

Sort of. In the early-to-mid 17th century, France and to a lesser extant Italy were awash with small volumes of books called "histoires tragiques" or "Tragic Tales." The first to write these short stories was François de Rosset, whose "Histoires tragiques de nostre temps" was re-edited nearly twenty times after its publication in 1617 and translated into German, English and Dutch.

Rosset's short stories were very gory, often depicted gratuitous violence, but always attempted to twist their endings for didactic conclusions. Stories ranged from "true" accounts of historical duels that ended in death, to fantastic accounts of young men unwittingly sleeping with a demon and subsequently being slaughtered when the demon reveals its true identity.

The morals of the stories are always very problematic. For example, a story about a son who knowingly kills his father in cold blood ends with the bland advice to respect one's family. These banal aphorisms never fully address the degree of the violence presented so gleefully in the stories themselves.

These baroque stories were fashioned on the type of literature written by Boccacio, especially his "Decameron," which presented a series of tales with no ostensible connection to each other. In the 16th century, Pierre Boaistuau and Matteo Bandello equally helped push the genre forward with their "Novelli" (translated by Boaistuau as "Histoires tragiques"). However, these precursor texts were not nearly as violent. Some would be "tragic" in the sense of ending poorly for the protagonists, but they did not revel in it, nor did their morals conflict tonally with the story.

Jean-Pierre Camus, who wrote in the mid 17th century, tried to address the specific problem that Rosset's texts posed; namely, that their unattenuated horror worked against their didactic mission. It was impossible, Camus argued, to teach something via the representation of violence, if the violence could not be controlled by the narrator himself. Camus' "Les spectacles d'horreur" (1630) were generically identical to Rosset's work, and in some stories even rivaled his pleasure in representing gore, but Camus always managed to make a fitting moral to that violence. Camus was also very religious - he was himself a Pastor - so many of the stories relied on theological arguments to explain horror. One common theme, for example, is that straying from the path of God leaves you at risk for gratuitous violence as you are not under the protection of the Lord.

Camus, however, was a little bit hypocritical, since all the while he is telling the reader that he wishes to correct the senseless horror of Rosset - he is still relying on Rosset's success to sell his own books. Camus models his horror stories on Rosset, while condemning the latter for writing horror stories! Even if Camus attempted to moralize and therefore instrumentalize violence, the central focus of his works is still making sense of evil through its textual representation.