Hey! A question I'm qualified to discuss! I'm not a historian but I have an MA in Literature and I've published peer reviewed academic papers on the subject of genre fiction.

There are two questions here: was Frankenstein the first horror novel and was Frankenstein the first Science fiction novel. Both of these questions are subjective but they nevertheless have different answers.

I'll start with horror: There is not a strong case Frankenstein was the first horror novel.

Some definitions: Novels are long form prose fiction. Which conveniently eliminates the need for us to consider whether poetic works like The Odyssey or Beowulf or any of countless Medieval romances or traditional ballads count as horror or science fiction.

The Icelandic sagas of the century do count, however. And quite a few of them contain monsters, ghosts, and the supernatural. I'm not an expert on the sagas, but the Saga of Grettir the Strong includes many battles against monsters on the island of Iceland. Monsters alone don't make a novel horror, however.

We'll define "horror novel" here as "a novel intended to frighten or shock its readers."

Which brings us to the gothic novel, of which Frankenstein was an example.

The first gothic novel was The Castle of Otranto by Horace Walpole, published in 1764. If we ignore gothic novels that don't have speculative elements (such as Anne Radcliffe's oeuvre), we're still left with several "horrific" gothic novels with supernatural elements predating Frankenstein. In addition to The Castle of Otranto, there's Vathek by William Beckford (1786), The Monk by Matthew Gregory Lewis (1796), and many others.

This last is an important one, first because it was wildly popular, and second because it shocked and terrified its audience. Samuel Taylor Coleridge (whose own poem Rime of the Ancient Mariner is a good example of "dark romanticism") wrote this in his review of The Monk:

"The sufferings which he describes are so frightful and intolerable, that we break with abruptness from the delusion, and indignantly suspect the man of a species of brutality, who could find a pleasure in wantonly imagining them;" (Critical Review, February 1797).

So, on what grounds could we call Frankenstein the first horror novel, but not The Monk, or to toss out another example, The Necromancer: A Tale of the Black Forest by Karl Friedrich Kahlert (1794)?

There are some reasons you might do so--a more contemporary setting, for example, but none that are unique among the entire canon of the gothic.

The question of when the gothic novel became the horror novel is not an easy or simple one to answer, but I'm not sure its relevant here. Frankenstein was written in the middle of the period when the gothic novel was most popular, and while it's of exceptional quality and originality compared to most of its peers in that genre, I don't think there's a strong case that it originated the horror genre. That honor either belongs to one of the Icelandic sagas, The Castle of Otranto (if the gothic novel = the horror novel), or to some later work.

The case for Frankenstein being the first science fiction novel is significantly stronger and, in my estimation, more persuasive. But it's hardly an open and closed case.To approach this question we need to agree on a definition of "science fiction" and there is no such agreement among literary scholars. In part, the difficulty lies in separating SF from other speculative genres such as fantasy and alternate history.

I'll propose two here:

Definition 1: Fiction featuring wonderous technology or scientific discoveries not extant at the time of the writing.

Definition 2: Fiction in which a primary concern is speculating on the consequences of a possible new technology or scientific advance. (You'll note that a great many works widely considered to be science fiction do not fit this definition--Star Wars, for example.)

There's no question that Frankenstein fits both of these guidelines decades before the works of Jules Verne and HG Wells popularized the genre. Frankenstein's approach to speculating on the consequences of a scientific discovery is certainly one thing that sets it apart from its contemporaries in the gothic genre and the dark romanticism movement.

By either of my definitions, and by most others one could propose, Frankenstein is a science fiction novel. No novel written after Frankenstein could therefore be considered the first. So we have to look at novels written before Frankenstein.

A True Story by Lucian (2nd century CE): Contains travel to outer space and contact with alien beings. Qualifies for definition 1, may not qualify as a novel.

Utopia by Thomas More (1616): Utopian literature is largely a separate tradition from science fiction until the 20th century--if it counts, it significantly muddies the waters for Frankenstein. In this case, it's hard to argue that the social and political organization proposed count as "science" in any meaningful way. I'll disregard other utopian novels (e.g. New Atlantis by Francis Bacon (1626)) on similar grounds.

The Chymical Wedding of Christian Rosencruetz by Anonymous (1616). Does alchemy count as science? Does an allegory for Rosicrucianism count as a novel? If so, it fits the definition 1.

The Man in the Moone by Francis Godwin (1638--posthumous). The method by which the narrator reaches the moon is unscientific (a ship pulled by geese). But he does discover a civilization there. Possibly fits definition 1.

Comical History of the States and Empires of the Moon by Cyrano de Bergerac (1657--posthumous): A satire, but the narrator does land on the moon and meet the beings who live there. Largely inspired by Lucian (above). Qualifies for definition 1.

Gulliver's Travels by Jonathan Swift (1726): Many, but not all, of the strange things Gulliver encounters fall more under the realm of fantasy than science fiction, but, to the extent that travel and exploration were scientific endeavors unto themselves, this book probably qualifies for definition 1.

I could go on, but the trend continues. Many, many books can be made to fit some definition of "science fiction" before Frankenstein, unless you use a definition in which the consequences of the science have to be the point of the novel. If you use that definition, Frankenstein has few, if any, contenders. But that definition fails to match with the public perceptions of science fiction.

However, as I said, it's all subjective. Literary genres do not have fixed definitions or unblurred lines. To me, none of the other possibilities for "first science fiction novel" feel quite as satisfying as Frankenstein does in that role. But it's not really a question with an answer.

[EDIT] Fixed some formatting issues and typos.

Horace Walpole's 1764 work The Castle of Otranto is generally accepted as the first horror novel. I write about this book in my response to the question "Are Gothic Buildings Supposed to Be Scary?" Walpole invented the Gothic genre, known for the elements of terror and wonder, with this text. It inspired many later writers, including Mary Shelley.

I'll leave the question of 'first science-fiction novel' to others. While there are examples of texts with sci-fi elements like space travel dating from Antiquity, the question is how do we define science fiction as distinct from fantasy and which novel was the first to contain these.

It's kind of a trick question because it's not about the genres of horror or sci-fi as much as its about the definition of a novel, which is much debated still. Shelley's "Frankenstein" is far from the first horror/sci-fi story, which can be traced back at least to the Greek & Roman writers of Antiquity.

Lucian's "A True Story" (2nd Cent Greek) is considered the first sci-fi novella, including aliens & space travel, however its a novella not a novel & it's as much parody & satire as sci-fi deliberately lampooning travel writers of the time by describing the most farfetched, absurd of journeys.

Horror stories can be traced back similarly to the the Greek era with "Hyppolytus" by Euripides (c428BCE) which is in fact likely an inspiration for Shelley, a tale of death, resurrection & consequences. However it's a play & derived from mythological material which is enough to disqualify it from being a "novel." Plutarch tells us in "Parallel Lives" perhaps the first ghost story of a murderer named Cimon who is himself murdered & returns as a spirit , however Plutarch is a philosophical writer & the story is likely an allegory representing the General Cimon's exile & return. Pliny the Younger (1st Cent CE, Latin) gives us the what might be the first haunted house story in the tale of Athenodorus Cananites who bought a haunted house, was visited by chained spirits & found relief from his haunting when he uncovered a hidden grave. But again, not a novel as the story is recounted in his letters (LXXXIII. To Sura)

So horror stories were certainly not a new invention by the "Gothic" era and even then Shelley didn't mark the beginning of that genre, she marks the beginning of the second generation. The first generation begins with Walpole's "The Castle of Otranto" (1764), while technically the first horror novel, it's publication involved some trickery on behalf of the author where he claimed it to be a translation of an older work which itself was a version of a story from the crusades. He later admitted that it was entirely a fiction he composed, so at the time it wouldn't have been considered a novel.

Following Walpole, came several other inspired by his work who each lose out on the 'first novel' title, Clara Reeve's "The Old English Baron" (1778) followed Walpole claiming to be a translation of an older work. Ann Radcliffe's "Mysteries of Udolpho" (1794) was published across 4 volumes. Goethe's "Bride of Corinth" (1797) perhaps the first vampire story, was poetry published in a magazine. The first vampire prose story was written in German in 1800 "Wake not the Dead" by Earnest Raupach & in English in 1823 (sometimes attributed incorrectly to Ludwig Tieck), but again this is a short story originally published in a magazine & in English in a collection.

So having skipped many other writers of the genre who all lose out on various technicalities, that brings us to Shelley's "Frankenstein" (1818). This takes the award of 'first horror/sci-fi novel" because it was: published in a single novel; was in a prose format; was published as a entirely original, not derived from mythological sources or a translation of an older work (either in truth or as a means of distancing the author from any potential backlash); was primarily created for entertainment purposes rather than allegorical or philosophical discussion (these themes might be present though not necessarily the primary function of the work).

So when it is claimed that "Frankenstein" is the first horror/sci-fi novel, it's is true but it's less about Shelley's invention of the genres & far more about the technical details of what constitutes a novel & the evolution of how fiction was published during the late 18th/early 19th centuries