As a great enthusiast of the Gothic novel (particularly in its early stage from the 1760s to the 1820s), I have noticed that the term "Gothic novel" was never actually used in this period. Sure, you had "Gothic romance," "Gothic tale," and "Gothic story," but these terms are never used to refer to works taking place outside "Gothic times" (i.e., the Middle Ages); thus, these designations would exclude most of the now canonical first-wave Gothic novels, including all Ann Radcliffe's novels published in her lifetime (none of which took place earlier than the sixteenth century), Matthew Lewis's The Monk (which takes place at an unspecified time during Spain's colonial period, since a character offhandedly mentions having landholdings in the West Indies), and Mary Shelley's Frankenstein (which is set in the 1790s). Furthermore, all of the seven horrid novels mentioned in Jane Austen's Gothic parody, Northanger Abbey, take place in either the seventeenth or eighteenth century. When was "Gothic novel" finally used to describe such works?
As far as earlier genre names go, I know that Charles Maturin used the term "Radcliffe-romance" to refer to his masterpiece, Melmoth the Wanderer (1820; another work that would not have been called "Gothic" at the time, as its earliest scenes take place in the seventeenth century). However, this seems to have been a term particular to him. Would a 1790's or 1800's reader of novels have even considered Matthew Lewis's very supernatural and irreverent The Monk as being in the same genre as Radcliffe's novels with their Scooby Doo plots (i.e., the explained supernatural) and their upholding of middle class propriety?
The Cambridge Companion to Gothic Fiction (2002) might be your friend. EJ Clery argues that "Gothic" referring to quasi-supernatural is a largely 20th-century description. His footnote:
1899: Henry A. Beers, History of English Romanticism in the Eighteenth Century (New York: Henry Holt) and Wilbur Cross, The Development of the English Novel (New York and London: Macmillan). It was established in Britain by Edith Birkhead in 1921, and in 1932 J. M. S. Tompkins followed suit with a chapter on “Gothic Romance.”
However the OED has two earlier attestations (added in 2007, so after the above source): Scott's Lives of the Novelists talks of "Gothic romance" in 1825 in reference to Walpole -- whose second edition of Castle of Otranto had the "Gothic story" appended to the title page. The OED notes difficulty in separating the supernatural "gothic" from the "medieval/times past" gothic usage that was contemporary to Walpole and the horror connotations it now has. Maybe more relevant, then, is this 1854 quotation:
G. S. Hillard Six Months in Italy (1854) II. x. 233 The shapes and conceptions of Gothic fiction—the sheeted ghost gliding from the churchyard..—the groan mingling with the wind that sweeps through the aisles of a ruined chapel.
In the chapters in the Companion as well as Christina Morin's (2018) The Gothic Novel in Ireland: 1760-1829, they seem to suggest that gothic fiction emerged as a response to Romances and other Historical Fiction (two genres that I've noticed occupy a prime place in 18th century history of the novel scholarship). Terry Hale's chapter in the Companion explores French and German c. 1790-1820. He uses roman noir, roman frenetique (coined in the 1820s), and Schaurroman, though he's less precise about the dates of the other terms.