It seems to me that improvisation has always been a major part of music making for all of history, but up until the jazz age, any music that wasn't written down was quickly forgotten (within a generation or two). Thus anything that was considered "serious" music got written down, and forced away from improvisation.
With the advent of recording technology it was suddenly possible for improvisations to be preserved for future listeners, thus "serious" jazz artists never had any reason to move away from improvisation, and instead developed it further than it had been before, until it reached levels of complexity that the average listener didn't have a taste for. The only reason we associate improvised solos with saxophones, swing-eighths, cycle of fifths progressions is because those were the musical idioms that were popular when it became possible to record improvised solos.
This seems intuitive to me when I think of it, but I don't have a good enough grasp of history to properly evaluate it. Can actual historians either confirm or refute this idea?
I don't agree.
I am not a Jazz specialist. I will deal with the issues based on music with a much older tradition that developed a writing and used improvisation before recording technologies were available. After that I get to Jazz.
any music that wasn't written down was quickly forgotten (within a generation or two)
Not always, no. Of course some has to be lost (not everybody wants to keep all music alive forever), and of course the music might change each generation, but musical traditions exist(ed) without musical writing. Having some form of musical notation is the exception and not the norm, most musical traditions have kept going without it. Music is part of a culture, things considered important are kept alive with or without writing (and the lack of recording technology didn't stop people in the past). Written music has a better chance of not disappearing completely, but that and not being forgotten are two different things.
anything that was considered "serious" music got written down, and forced away from improvisation.
When it first appeared, music writing presented a way to standardize the music used in religious ceremony in the Western world. Sure, it was a way to keep music alive, but that was not the issue they were tackling. They tried to get people into having a common repertoire. Religious music was pretty serious, people kept it alive without musical writing before this time. It's just there was a lot of music and standardization was pushed. People were learning the same music, notation was an aid for them to remember it.
Did musical notation reduce the amount in improvisation? In some ways, yes. If you are writing down each and every note, and you ask people to stick to those, you just remove a lot of improvisation (this was not the case at first). However, that doesn't mean they wanted to ban all improvisation from music. We get details on improvisation techniques from medieval treatises, people were taught how to add another melody to pre-existent chants. Medieval and Renaissance musicians learned how to improvise a counterpoint over a cantus firmus (both for religious and secular music).
Improvisation has been a big part of musical education, AND people worked hard on developing a more precise notation system (these things happened at the same time). It was serious music, people had a writing system and spent time learning it, and they also learned how to improvise. One thing does not replace the other.
Baroque music? Required improvisation (good luck trying to be a harpsichordist without being able to improvise). Great singers had to learn their music for operas, and they had to be good at coming up with clever ornaments and variations for their arias. Classical music? You had to create your own cadenza, and play clever (or at least not offensive) things in the fermatas.
Liszt, Mendelssohn, Paderewski, and Anton Rubinstein were all improvisers. "Piano duels" were a thing for a while, and pianists improvised.
With the advent of recording technology it was suddenly possible for improvisations to be preserved for future listeners
If anything, the decrease of improvisation in classical music starts close to the time recording technologies emerged!
thus "serious" jazz artists never had any reason to move away from improvisation
Who are these "serious" musicians? Who are the "not serious ones?"
Wait, what exactly is improvisation?
Contemporary accounts tell us Chopin never really performed his works in the same way twice. That music was pretty serious stuff for him and others. Is that improvisation? Tempo, phrasing, articulation, pedaling, rubato... He was VERY particular about the details written down, spent very long periods working on small details (George Sand noted she would write whole books while Chopin kept working on the same measures). Chopin also used to change manuscripts at the last minute, creating several different versions of his works. He even made modifications to the sheet music his students were working with during their lessons. Was he improvising?
Why did classical musicians move away from improvisation? Were they forced to do so because of their lack of a way to record their improvisations? No, they were not trying to do that. The concert music tradition started focusing on things other than improvisation WAY BACK. Notation was a big part of that.
What reasons are there to move away from improvisation? Why did they notate more and improvised less? The emphasis on structure, detail, and complexity is a possible reason. Different goals, different musical intentions. Maybe improvisation was left as an expressive device, a way for individual performers to be themselves, while still making music with specific objectives and characteristics. Some times it could have been a way to fill the blanks, if you wish, but it was a highly regarded skill, integral to music.
How did we get into structure, detail, and complexity?
Literacy changes things, writing it is a very powerful technology. When musical notation appeared in the middle ages, it changed the EXISTING music and changed the way musicians started composing from then on. People were after standardization, order. Modes (not the Jazz ones) were developed to have a way to catalogue things, to be able to say what music can be easily combined with some other music. You want people to be able to work with a big predefined repertoire, you now have a musical bureaucracy and theory was developed to make it work smoothly.
Musicians absorbed that. It changed the way they thought about music. Let's go to a point in time in which notation is readable for us.
What were musicians working with back in ye olde days? Let's say the 13th century. It gets difficult to try to coordinate several people improvising this kind of thing. How do you improvise aurally interesting music that adheres to peculiar aesthetic ideas of consonance/dissonance, rhythm? How do you coordinate one singer imitating what the other just did without things getting terribly repetitive? Lyrics, they all are singing texts that are important, you don't want everybody going his own way because it becomes complete gibberish.
Is is possible to improvise that? I don't know, but sounds difficult. It gets easier if you write it down first. If that's the kind of music you want to make, that is.
They didn't have particularly many cognitive tools to work with simultaneous sounds, or even with rhythmic complexities. They dealt with intervals, not static individually recognizable chords they could agree on before hand. Ensemble music was not written vertically in a score as we are used to now, people were working with individual melodies. Melodies, yes. Lots of them. Compare that structure to what you find in a Jazz trio.
Let's move on to the 15th century. More tools to deal with the rhythmic complexities, more solid ways to deal with scales, but good luck trying to improvise anything with the level of complexity of that Missa. Of course, it's not that they had to make that kind of music, but it's what many were into. That's what they grew up with, it had been this way for a while. This is what great musicians were expected to do.
Medieval and Renaissance musicians were not deaf and I am sure secular music was improvised without issues at the pub, but it was a different kind of music. Singers learned how to improvise stuff, but other kinds of stuff.
Seventeenth-century Venice. People loved their melodies, spent a lot of time polishing them. They provided detailed notation for melodies, and enough information to create some form of accompaniment for them, frequently using short hand notation. Sounds awfully similar to a lead sheet. People HAD to improvise, they still liked their structure and were VERY fond of texts. The text actually dictated how you were going to make the music sound.
Melodies, structure, order, detail... People were inspired by rhetoric and that kept going for a long time. And all that super specific tradition coexisted with improvisation for a very long time.
Here's and example of a man who was a revolutionary. He was into the usage of repetitive patterns instead of only independent melodies. Six instruments have predefined parts. You can change them a little in real time (well, the 3 solists, mostly), but this counterpoint is a very difficult thing to improvise in groups. A harpsichord/lute/organ and maybe another instrument would improvise most of what they would be playing.
Different goals, different resources, different music. They improvised (J. S. Bach was a contemporary of Vivaldi, and was known for improvising crazy things), their theory classes were meant to teach both how to improvise and how to work on larger scale works with a lot of emphasis in structure and order.
From then on large scale order and structure became super important in the classical world for a while (the Germans kind of took over). They wrote some massive musical things with more and more musicians. People had already a lot of handy cognitive tools, and developed more. They built with those, they built big. They actually had to invent stuff to make sense out of such big bunches of sound. People got obsessed with studying the past, and "museum musicians" were trained to be something close to curators. By the time Jazz was starting, improvisation was diminishing in the classical world.