I'm thinking specifically of books like 'Call of Cthulu', 'Jekyll & Hyde' and 'Frankenstein' which both have the main plot told by a character recounting what they heard from a different character who witnessed the events, aswell as stories like Dracula with unconventional framing. When did this style become popular and when was it replaced with modern story framing? Is it limited to horror?
Sorry if this is the wrong subreddit for this, I don't know if there's a good 'AskLiterature' sub or anything.
There is actually two separate literary influences going on here. So let's take them separately and then show how they came together.
Frankenstein; or the Modern Prometheus (1818) consists of a series of nested narratives in the form of a collection of purported documents and accounts. The form of diary entries, letters, etc. is something that would have been very familiar to audiences at the time, who would be used to receiving information in the form of long, detailed letters - essentially epistolary novels - diary entries, written confessions and narratives, etc. So the "found document" format would have increased the verisimilitude of what would otherwise be an absolutely fantastic and unbelievable tale, grounding it in reality - and allowing for lots of literary play as far as the reliability of narrators, letting the reader piece together clues from disparate documents, etc.
It also makes a convenient format for a "nested narrative" - where a series of letters on encountering a stranger leads to the stranger presenting their own tale which is then set down, etc. etc. Nested narratives are very old, particularly the idea of a "frame tale" - like the One Thousand Nights and a Night, where the "frame story" is Scheherazade needing to keep the Sultan entertained with a new tale every night. It makes a convenient narrative construct for telling an episodic tale, or even incorporating new stories into an existing frame (for example, some versions of 1,001 Nights might incorporate material from the Odyssey).
So neither of these were invented by Mary Shelley; before she came around it was a very popular format in The Manuscript Found in Saragossa and the Decameron. What she did was set her stamp on what became Gothic horror; the very convincing format of the series of found documents worked extremely effectively when done right - hence why Bram Stoker did the same basic thing in Dracula (1897).
Robert Louis Stevenson was certainly aware of both the "found document" and "nested narrative" format, but his strongest influence was on the latter. The Strange Case of Jekyll Hyde, Dr Jekyll and Mr Hyde (1886) can be seen as a series of related stories that amounts to a novel, and this really makes sense when he was deliberately working on the format of basically "the frame story without the frame" in books like New Arabian Nights (1882), More New Arabian Nights (1885), and Island Nights Entertainments (1893).
One of the writers that Stevenson strongly influenced was Arthur Machen, who created - you guessed it - a couple of frame stories in "The Great God Pan" (1894) and the nested narrative/Arabian Nights-esque horror novel The Three Imposters (1895), which critics and admirers alike noticed was very "Stevensonesque" - to the point that some claimed Machen was copying Stevenson.
What Machen did that Stevenson did not do was seed several of his stories with the implicated of a shared mythology. This was never fully developed in a single volume, but those who really enjoyed Machen's fiction picked up on the connections between disparate stories - like fragments of a lost mythology pieced together from different books. The exact same kind of piercing-together-of-evidence that worked with "found document" narratives.
Machen did use found documents, most notably in "The White People" (1904), but usually it was embedded in a narrative - i.e. the story would begin with a straightforward narrative, and then the characters would discover the document, which would then form the story-within-the-story; think of it as a frame narrative with only a single episode (chain those together and you could have a novel!)
Cue H. P. Lovecraft. At this time, pulp fiction was generally much more straightforward, with relatively simple narrative structures. But Lovecraft was very influenced by Arthur Machen, especially the idea of the artificial mythology that Machen had hinted at, and which Lovecraft would develop to a much greater extent. "The Call of Cthulhu" is probably Lovecraft's most complicated story from a frame standpoint, because of the embedded narratives, but the key thing is that they are narratives, organized into distinct chapters, not found documents. Like Machen and Stevenson, Lovecraft wasn't above using a found letter or diary entry to add verisimilitude, and "The Call of Cthulhu" is presented as being inspired by a collection of found documents - but the documents themselves are not often presented; Lovecraft does a lot of summary and abbreviation - he is, in short, presenting this as a narrative of a case compiled by notes, rather than the collection of documents themselves.
Which wasn't exactly common for him or for horror fiction in general, but you can see how the influences he was drawing on got to that point. Lovecraft was striking a balance between a collection of found documents communicating a nested narrative and the more straightforward format of regaling a horror story as it happened.