If you look at interviews from the time, they tend to emphasize the counter-culture resonance of the books, or how they are Modern in some way, although I am fairly skeptical of this; the explanations come off as fairly ex post facto, trying to retroactively fit a phenomenon slightly out of the norm into current events; for example:

No youngster is going to believe in a beautiful knight on a white charger whose strength is as the strength of 10 because his heart is pure. He knows too much history and/or sociology, alas, to find knighthood enchanting in its feudal backgrounds and to dream of Greek heroes and of gods who walked the earth. But give him hobbits and he can escape to a never-never world that satisfies his 20th century mind.

Tolkien was in reality a literary conservative reaching for the deep past (he disapproved even of Shakespeare), and it isn't hard to find that in his books. While groups like The Beatles and Led Zepplin were enthusiasts, this enthusiasm was not reciprocated by Tolkein (see my previous answer here for a little on this).

The books had relatively steady sales although not pop-culture level. Where the books suddenly became huge in the US was a (kind of) pirated Ace paperback version.

Ace was one of the biggest publishers of science fiction at the time. Ace had originally started as a line of comics (generally mystery, but some romance and western tossed in), published by Aaron A. Wyn (a Russian immigrant) before they expanded to book publishing in the 1950s and eventually phased out their comic line.

The book expansion was mainly due to the editor, Donald A. Wollheim, who was already responsible for one of the earliest sci-fi book collections, and had been trying to wheedle Wyn into book publishing; he had in fact been in negotiations with Pyramid for a new job, but a call for references got redirected incorrectly and Wyn found out about Donald's intentions to jump ship. He immediately went and offered the publishing job, so Ace Books was born.

Ace Books still kept with mystery / western standards although started to insert sci-fi early (again, this was Wollheim's passion) and that slowly ended up dominating their lineup, publishing leading authors like Ursula Le Guin and Roger Zelazny.

By 1964, sales of Lord of the Rings were respectable but not pop-culture-phenomenon level; in particular, there was no paperback version (Tolkien did not feel like his book was "mass market"). Wollheim, while not a fantasy specialist, recognized that the books were something special, and called Tolkien in that year asking about publishing the books as paperbacks. He was rebuffed (something about paperbacks being "degenerate") which offended Wollheim, being enmeshed in the paperback business and knowing how much popularity the format could bring. He eventually realized a "loophole" in the copyright law -- specifically, as this was before the Berne Convention of the late 70s, this was back when you had to declare copyright in a particular country and also intentionally renew it (issues like this were why the original Night of the Living Dead ended up being out of copyright). The books being sold in the US were simply "published in the UK" and they were popular enough that Houghton Mifflin had violated import limits and had (apparently) handled US copyright renewal incorrectly.

The snub plus the copyright situation led Donald to go ahead with what are now infamous "unauthorized" versions of Lord of the Rings. The books went from respectably-good-sellers to a phenomenon. (Tolkien and his publisher was already in the process of making a revised version that could have the copyright arranged correctly -- they found out Ace was putting out their version while in the process.)

The Ace version came out first and sold 100,000; Tolkien started to let fans to know about the unauthorized status of those versions.

I am now inserting in every note of acknowledgement to readers in the U.S.A. a brief note informing them that Ace Books is a pirate, and asking them to inform others.

Ballantine published the "real" version and there ended up being great pressure on Ace, with some places refusing to sell their version (even though it went for cheaper, 75 cents per book versus 99). There was enough pressure that in February 1966 Wollheim made a royalty agreement and also agreed to not making any further printings (not under legal obligation! ... the loophole was real).

The legal fuss ended up creating extra publicity leading to increased sales of both, and college student word-of-mouth enthusiasm was enough to push 1966 sales into the stratosphere. Wollheim suffered professionally, even though he was technically legally in the right; he was nominated for but never won the Editor Hugo, allegedly because of bad blood remaining over the incident. It is hard to say if the piracy was necessary for the second life of the book; really the big burst of publicity came from the giant fan campaign afterwards, but it comes too much into what-if territory to ask if the already strong word-of-mouth would have been good enough.

...

A Companion to J. R. R. Tolkien. (2020). United Kingdom: Wiley.

Drout, M. D. C. (2007). J.R.R. Tolkien Encyclopedia: Scholarship and Critical Assessment. United Kingdom: Routledge.

Knight, D. (2013). The Futurians. United Kingdom: Orion.

The Lord of the Rings: Popular Culture in Global Context. (2006). United Kingdom: Wallflower.

The Lord of the Rings was started in the late 30s as a more serious sequel to The Hobbit. It was finally published - somewhat reluctantly - in the late 50s.

Tolkien started writing a sequel to The Hobbit in 1937. Between academic work, revisions, and an inconvenient world war, the book was't finished until 1949.

Tolkien fell out with publisher George Allen & Unwin so the book did the rounds of other publishers and eventually ended up on the desk of William Collins, who offered to print it as long as it was split into three volumes. Because it was an unlikely prospect, Collins didn't want to commit to printing a large book that might not sell. So The Fellowship of the Ring tested the market in 1954, and the market proved just about large enough to support the remaining two books.

At this point the faux-medieval imaginary worlds we associate with fantasy weren't considered a separate literary genre. "Fantasy" meant imaginative unrealistic fiction. Stories were published in pulp magazines and tended to have scientific themes. But there was also a minor side-tradition of heroic fantasy and horror, often with orientalist and/or dark occult undertones, sometimes based on human protagonists being displaced to other planets.

Edgar Rice Burroughs was possibly the most famous early heroic fantasy author, starting a run of stories in 1912 about an imaginary Martian civilisation which was an original and apparently appealing mixture of muscular and oriental tropes. (Burroughs actually became a franchise, with ghost writers producing stories under the name.)

Other notable fantasy-ish books included Heinlein's Magic Inc, a strange collision of occultism and demonism with American small business lore, L Ron Hubbard's Slaves of Sleep, which is fairly traditional Arabic orientalism, and Perelandra by Tolkien's friend CS Lewis, which depicts an exotic fantasy utopia on Venus.

So there was a niche undercurrent of exotic and fantastic fiction, but it was far outside of the mainstream. The content and mood tended to be inspired more by Hollywood tropes - especially by stock American Hollywood hero figures - than by scholarly research into mythology and linguistics.

One major exception was the Harold Shea series by L. Sprague de Camp and Fletcher Pratt, which explicitly used various mythologies as its starting point. But unlike LOTR the focus was entertainment and slapstick humour, and the mythologies were included for colour rather than scholarship.

LOTR dropped into this backstory and made some small ripples, with more interest - both positive and critical - from academics and literary figures than the general public.

In the UK Collins made a small profit on their editions, and in the US the books were published by Houghton Mifflin, to minor and very limited acclaim. LOTR won the International Fantasy Award in 1957, which was less prestigious than it sounds, partly because it was the last ever International Fantasy Award, and partly because - as explained above - the word "fantasy" meant any kind of imaginative fiction.

The book might well have been forgotten and/or relegated to cult curiosity status, if the CEO of US Publisher Ace hadn't tried to steal it. It was decided that because the copyright wasn't formally registered in the US, Ace were free to publish their own editions. Without royalties to Tolkien.

At the time, Ace were one of the biggest pulp/SF/"fantasy" publishers in the US, with a significant readership and a huge national distribution network.

Houghton Mifflin, meanwhile, specialised in textbooks and non-fiction. Fiction was something of a sideline.

So quite suddenly LOTR had mass market distribution in the US, with access to tens of thousands of potential readers who already had a taste for "fantasy" titles.

While it didn't immediately become a best-seller, it started to make an impression with a receptive public, and also began to be promoted through the most effective of all possible marketing media - word of mouth and personal recommendation.

Ace were soon put in their place by lawyers, and the book moved to Ballantine - ACE's biggest competitor, who also had mass market distribution and a history of fantasy and other imaginative titles. Tolkien even included a note pointing out that the Ballantine edition was official and approved, and this effectively took the ACE edition off the shelves. (The ACE edition had a very limited run, and copies now change hands for $450.)

Of course this doesn't explain why the book was popular in the first place. Unsurprisingly there's been huge interest in this among critics, and equally unsurprisingly there are no definitive conclusions. Here are some of the more popular suggestions.

Compared to previous "fantasy", the differences are obvious. The sheer scale and detail of the world building is unique and only modern multi-author franchises like Star Wars and Star Trek begin to approach it.

Almost every element and character has an important and detailed backstory. The Hobbit and the Silmarillion are further views into the same world which add even more detail. There are also maps, poetry, calligraphy, and an invented language. (Tolkien originally wanted The Silmarillion to be published with LOTR, but a volume of imaginary mythology was a project too far for his publishers.)

The mood was also unique - a dream-like reinvention of familiar mythologies set in a reimagined medieval magic-friendly medieval world. It captured in equal parts nostalgia for a small-c conservative pre-industrial green utopia (with evil wizards, but never mind...) and fantasies of magical omnipotence.

It would take a separate article to to detail how the sixties hippie occult revival seized on this, but it was certainly a factor. There also isn't space to detail how the mood was influenced by the stylised pastoral naturalism of William Morris and the English Arts & Crafts Movement and the brightly coloured adolescent medievalism of the Pre-Raphaelites, both of which gave the book added cachet with some of the UK's literary set.

Perhaps most importantly - the heroes are ordinary everyday people. They are not Hollywood bare-knuckle alphas who use their fists and their wits to outpunch and outsmart their evil enemies while cracking jokes and rescuing the proverbial girl. This makes the books less popular with a small audience, but more relatable to a much wider audience.

Also, dragons. Who doesn't love dragons?

To summarise - it's not an exaggeration to suggest LOTR invented modern fantasy. There was nothing exactly like it until it was published. Since then there have been thousands of imitators vying to populate their worlds with battle scenes, dragons, wizards, and assorted cloak-wearing heroes of various talents and orientations. But none have the scholarship or inventiveness of LOTR. And none have sold nearly as well or made such a mark on culture.