The Cornish have what are probably the most well-known mining spirits - called knockers or bucca. These are almost certainly the source of the North American tommyknocker, an entity that is nearly the same and is popularly attributed to the Cornish immigrants, often called 'Cousin Jacks'. Although the tommyknockers began as immigrant lore, they are unique in the way they became the common property throughout the mining West with stories told by miners regardless of ethnic background.

The tommyknocker provides the clearest path to providing an explanation of how the tradition began, namely this was an example of immigrant folklore traversing the Atlantic, but in this case, the tradition did not whither the way most other did, but rather it thrived and diffused among other miners.

The idea that there are spirits in the mines is international. Quoting now from my forthcoming book The Folklore of Cornwall: The Oral Tradition of a Celtic Nation:

Perhaps wherever miners went underground during pre-industrial times, they imagined the eerie environment populated with supernatural beings. Germans referred to their own similar spirits as Kobolde or Wichtlein; for the Welsh, they were Coblynau; in Bohemia they were known as Haus-Schmiedlein; in Malaya as Chong Fus and in the Andes as Muquis. For the English, the mine spirits were goblins or dwarfs. Although the Cornish called them knockers or knackers, sometimes they used indigenous terms for supernatural beings: Bucca, Bogle or Spriggan. ‘Knocker’, after all, was something of a generic word. According to an 1896 article in Folklore, Burne describes evidence that the miners of Staffordshire had a tradition of ‘mining dwarfs’ that behaved much like the Cornish entities and were, in fact, called ‘knockers’. Indeed, it seems that the name ‘knocker’ was widespread. This is not surprising since the word refers to noises often associated with underground supernatural beings: that is, they ‘knocked’ as timbers creaked and groaned and as the forces of nature made themselves known in other audible ways. Although miners elsewhere in Britain commonly used the name ‘knocker’ for underground spirits, the word quickly became linked in the popular mind with the Cornish who were prominent in mining and consequently came to shape much of the industry’s folklore.

It is not easy to explain how these types of traditions begin. Again from my book: 'These mining spirits are reminiscent of piskies [Cornish fairies or elves] and similar supernatural beings aboveground, dangerous but nevertheless capable of generosity towards people of good character. Unlike the piskies, the knockers commanded the subterranean world of the mine, a treacherous and eerie place.' So this is a partial, unsatisfactory answer to your question, namely that knockers are merely fairies/elves/piskies that appear underground, but that just pushes the question further back, leaving unanswered the questions as to the origin of the larger tradition in the full range of supernatural beings. That question remains unanswered and unanswerable thought wild speculation in rampant.

The earliest mention of knockers appears in (again from my book):

In 1851, Devon-born Charles Kingsley (1819–1875) published Yeast: A Problem, in which his character, Tregarva, answers Sir Lancelot’s question, ‘What are the Knockers?’. Tregarva, which British readers would recognize as a Cornish name, replies that 'They are the ghosts, the miners hold, of the old Jews, sir, that crucified our Lord, and were sent for slaves by the Roman Emperors to work the mines, and we find their old smelting-houses, which we call Jews’ houses and their blocks of tin, at the bottom of the great bogs, which we call Jews’ tin.'

An older reference to this sort of underground miner appears in two sources by Georgius Agricola (1494–1555), describing German tradition: 'His De Animantibus Subterraneis Liber identifies a spirit he called Cobalos. He claims the Greeks shared these sorts of entities, which were gentle, full of laughter and eager to appear to be working.' Agricola describes the following:

They are called little miners, because of their dwarfish stature, which is about two feet. They are venerable looking and are clothed like miners in a filleted garment with a leather apron about their loins. This kind does not often trouble the miners, but they idle about in the shafts and tunnels and really do nothing, although they pretend to be busy in all kinds of labor, sometimes digging ore, and sometimes putting into buckets that which has been dug. Sometimes they throw pebbles at the workmen, but they rarely injure them unless the workmen first ridicule or curse them…. The mining gnomes are especially active in the workings where metal has already been found, or where there are hopes of discovering them and cause them to labor more vigorously.

I published on this a quarter century ago, but my thoughts and research have progress since then, so my forthcoming book is the better source on this. Here is the older reference: Ronald M James, ‘Knockers, Knackers, and Ghosts: Immigrant Folklore in the Western Mines’, Western Folklore, 51:2 (April 1992), pp. 153–76.

I hope this helps.

edit: source for Kingsley: Charles Kingsley, Yeast: A Problem (London: J.M. Dent and Sons, 1851), p. 255.

Other sources:Charlotte S. Burne, ‘Staffordshire Folk and their Lore’, Folklore, 7:4 (December 1896), p. 371. See also, Richard Suggett, ‘The Fair Folk and Enchanters’, an article dealing with Welsh folklore from Young and Houlbrook, editors, Magical Folk, p. 142.

See also: Lydia Fish, ‘The European background of American miners’ beliefs’, in Folklore Studies in Honour of Herbert Halpert, Kenneth S. Goldstein and Neil V. Rosenberg, editors (St John’s, Newfoundland: Memorial University of Newfoundland, 1980).

For Agricola: Georgius Agricola, De Animantibus Subterraneis Liber (Basiliae: Apud Frobenium et Epicapium, 1549), pp. 77–79. See also Agricola’s De Re Metallica, translated by Herbert Clark Hoover and Lou Henry Hoover (New York: Dover Publication, 1950), p. 217.

For tommyknockers, see also Wayland Hand’s ‘The Folklore, Customs, and Traditions of the Butte Miner’, see his ‘Folklore from Utah’s Silver Mining Camps’, Journal of American Folklore, 45 (1941), pp. 132–61; and see Hand, ‘California Miners Folklore: Below Ground’, California Folklore Quarterly, 1 (1942), pp. 127–53; and Bancroft, ‘Folklore of the Central City District, Colorado’; Baker, ‘Echoes of Tommyknockers in Bohemia, Oregon, Mines’.