It seems as though the concept album was largely confined to the 70s – The Six Wives of Henry VIII (1973), Journey to the Centre of the Earth (1974), Captain Lockheed and the Starfighters (1974), Animals (1977), Jeff Wayne's Musical Version of the War of the Worlds (1978) and The Wall (1979), to name some of the most prominent examples. Yet, looking at the 80s, I can't really think of any concept albums of note. Why, then, did they take off, and why haven't they made a major comeback?
1/2
Firstly, let's talk about what an album is. So, the word 'album' originally comes from the idea of the album implicit in the term 'photo album' which (in the pre-internet age, at least) is a book that holds separate photos in a single package. The original albums were albums in this sense: they were pre-packaged collections of single 78rpm discs, a format that, at least in its popular form, usually only held 3-4 minutes of music on each side. Together, with a set of 78rpm discs packaged together, you had an album. The first concept album inevitably dates from this period, not the 1970s. For example, Woody Guthrie's Dust Bowl Ballads, from 1940, was a folk concept album revolving around the experience of Depression/dust bowl-era 1930s in Oklahoma (song titles including 'The Great Dust Storm (Dust Storm Disaster)', 'Dust Bowl Refugee', 'Dust Bowl Blues' and 'Dusty Old Dust').
When the 12" 33rpm record became popular in the 1950s - when you could fit all of an album on one disc - concept albums soon followed, such as In The Wee Small Hours by Frank Sinatra, from 1955 - as one of the first pop albums to be released on one 12" 33rpm disc, it was also therefore one of the first pop concept albums to be released on one 12" 33rpm disc (seeing as the songs had a theme of late-night loneliness tying them together, like the title track).
However, in the mid-1960s, with the rise of multitrack recording technology, you get a different kind of conception of what an album is to what had previously come. To some extent, in the 1950s, the attitude towards the album was that it was simply a collection of performances. However, from the 1960s, people started viewing the album not as a collection of recordings of performances, but a thing in of itself; the album wasn't an avenue for you to hear the Beatles performing 'A Day In The Life' - it was no longer simply musicians performing, caught on tape, but a physical artifact - a piece of vinyl - deliberately designed to be received as art. The Beatles and their producers have generally denied that Sgt Peppers' Lonely Hearts Club Band (released in May 1967) is a concept album in the way that, say, Tommy is - obviously, the first three songs are sort of thematically linked, but then it goes in other directions - but it absolutely was a concept album. With its sounds being electronically manipulated in a way that would mean the Beatles couldn't actually sound like that in real life, and in its emphasis on the album being a physical artifact - the ornate packaging, the effort put into the album cover, the loop in the runout grooves, for example - the concept of Sgt. Peppers, in a way, was of a new conception of the album as an artwork in of itself. An enormously popular new conception of the album, seeing Sgt Peppers' enormous success.
The albums you mention - by Rick Wakeman, Pink Floyd, etc. - historically follow in the wake of Sgt Peppers, which - along with some other ambitious albums of the period, such as Pet Sounds - inaugurated a new era of 'rock' music which distinguished itself from run of the mill pop by its sonic ambitiousness, and its desire to progress artistically, to do something new.
At this point, after the success of Sgt. Peppers, which had elements of 20th century (classical) art music forms like musique concrete in it, several artists started to combine rock music with other musical forms that were typically longer than the 3-4 minute pop song. For example, The Moody Blues released Days Of Future Passed in November 1967, which features orchestral interludes by the London Festival Orchestra, and which clearly aims at a sort of classical form in the parts of the album without orchestra. The Moody Blues, and other such ambitious, largely British rock bands, were well-received, and the impulses of groups like Yes and Genesis and ELP came together in a genre called progressive rock, which aimed to progress from Sgt Peppers' starting point towards a music which aimed towards the ambitiousness and complexity of some of the forms of classical music. The point at which progressive rock really gets going commercially and artistically is basically on the turn of the 1970s.
With that in mind, let's talk about the concept album. Firstly, there are a variety of different kinds of concept album. At the most basic level of the concept album is the idea that there is a thread running its way through these songs - these songs have been collected together for a specific purpose.
One of those purposes is a consistent lyrical theme - see Dust Bowl Ballads or In The Wee Small Hours, or, for example, Little Deuce Coupe, the 1963 album by the Beach Boys that's all about cars. Animals which you list above, fits this category. Another is that there is a consistent use of particular musical sounds or events on the album. So, for example, the band Self released an album in 2000 called Gizmodgery which is a concept album because everything on the album is recorded with toy instruments (hear their version of 'What A Fool Believes').
At a slightly more advanced level of conceptual-ness is an album that has a sort of progression of situations through the album - a song cycle where the songs are meant to be presented in a specific sequence. Pink Floyd's Dark Side Of The Moon goes from birth (the infant cries and heartbeats at the very start of the album) to death (a song called 'Eclipse'), for example, and there's particular ways in which the music of the different songs on the album fit together thematically. But other albums might have a sense of progression in this kind of way without necessarily having a specific theme - there might be some elements of the kind of motifs and musical structures that you might see in a symphony, mixed with pop music forms (though, usually, with rather more widdly guitar and burping synth than the average classical symphony).
And then there are albums that specifically do 'rock opera', in the sense that the albums are storytelling narrative, told through song - quite like an opera or musical, except in rock album form. Tommy by The Who, from 1969, was self-consciously a rock opera, described as such in its promotional material (and later presented in film musical form, i.e., with Elton doing 'Pinball Wizard'). You can often imagine rock operas performed as musicals, and in fact Andrew Lloyd Webber and Tim Rice's Jesus Christ Superstar was first a rock opera album, released in 1970 with Ian Gillan of Deep Purple playing Jesus (here on 'Gethsemane'), before it was first performed live in 1971. War Of The Worlds, with its narration is very clearly a rock opera. And The Wall too, with its somewhat abstract but fairly clear narrative of a rock star becoming progressively more alienated as explicated in the film version.
So the various types of concept albums took off, essentially, because in the wake of Sgt Peppers there was a young, but rapidly maturing baby boomer audience who wanted to believe - in the wake of Sgt Peppers that rock music could continue to be important, that it could be High Art. In wanting to be High Art, the rock musicians emulated the forms of High Art they saw in socially-esteemed classical music - symphonies and operas.
So the explosion of concept albums in the 1970s was predicated on the belief that rock music was reaching towards High Art, and that the way in which to reach towards High Art was to emulate classical forms. These beliefs did not last in the discourse around popular music. By the late 1970s, critics and younger audiences championed punk, which self-consciously rejected any prog rock pretensions to High Art (John Lydon in his autobiography Rotten talks about how he became a member of the Sex Pistols largely because they saw him in his famous 'I hate Pink Floyd' t-shirt; he also claims that "Yes or any of that stuff...was too arty, distant and remote, all about 6/4 masturbation"). Punk largely championed the 3 minute single, and musical simplicity - it was part of the myth of punk that they could barely play their instruments. Not necessarily because of punk, but the late 1970s saw the bigger progressive rock bands ultimately move away from forms and sounds that aimed for classical music. Genesis transitioned from Peter Gabriel as lead singer to Phil Collins as lead singer, and eventually started making fairly straight pop music ('Follow You Follow Me'). Trevor Horn of The Buggles produced an album for Yes, which ended up having a new wave style hit in 'Owner Of A Lonely Heart'. Pink Floyd's 'Another Brick In The Wall, Pt. 2' might be from a concept album, but it has a disco beat.
Follow-up questions: What caused the rise in their popularity in the first place? Was there a particular artist or album that started a trend or marked a significant cultural change?
Also, FYI, concept albums still exist (and seem to be popular among smaller groups of fans) in the hard rock/metal world.
Tool - Lateralus (2001)
Tool - 10,000 Days (2007)
Keldian - Outbound (2013)
Thrawsunblat - Wanderer On the Continet of Saplings (2013)
Thrawsunblat - Metachthonia (2016)
Woods of Ypres - Pursuit of the sun & allure of the earth (2004)
Anubis gate - Horizons (2014)
Brendon Small - Doomstar Requiem (2013)
Those are just the ones I can think of off the top of my head, that I've listened to, and that I believe qualify as concept albums because they are tied lyrically, musically, and thematically from start to finish, including ALL tracks. If we loosen the definition of "concept album" a bit, the list of candidates from the 80's or later gets very long very quickly.
However I do agree that they seem to have lost the universal cultural appeal of your examples from the 70s. And I would also like to know why.
Yet, looking at the 80s, I can't really think of any concept albums of note.
There were a lot of bands that today would be called "power metal" or "progressive metal" doing concept albums in the 80s. The most prominent being:
Queensryche - Operation: Mindcrime, released in 1988, is a rock opera with a clear story progression from track to track about a dystopian near future. Rage for Order (1986) and Empire (1990) also have strong concept album elements, although not every song on those albums are tied together.
Iron Maiden - Iron Maiden were incorporating some thematically related songs as early as 1982's Number of the Beast, which continued into Powerslave (1984) and Somewhere in Time (1986). Seventh Son of a Seventh Son (1988) meets all the criteria for a concept album, with every song based on Orson Scott Card's novel Seventh Son.
King Diamond - Abigail (1987), thematically centered around ghosts and hauntings.
Helloween - Most of their albums have some concept elements, especially Keeper of the Seven Keys Pt 1 (1987) and Keeper of the Seven Keys Pt 2 (1988), which were originally intended to be released as a double album, and finally were in 2010.
Outside of strictly heavy metal, we've got:
Blue Oyster Cult - Although Imaginos (1988) is credited to BOC, it's probably better described as longtime producer and lyricist Sandy Pearlman's project tying together many of the repeating themes in previous albums by the band.
Planet P - More or less a one man band (Tony Carey), Planet P's second release, Pink World was released as a double album (On pink vinyl!) in 1984. It's about a highly dystopian post nuclear apocalypse society contained inside an area called "the Zone".
One can debate how "notable" some of these albums are outside their genre. Pink World is somewhat obscure, and Imaginos suffered from distribution and record label problems, but Operation: Mindcrime and Seventh Son of a Seventh Son both hit the mainstream charts (Seventh... was #1 in the UK), and Mindcrime was certified platinum, Seventh gold by the RAA.
Concept albums didn't disappear in the 80s, they were just in a different place.