Musicians previous to Bach's time would generally be expected to play variations , often over a "ground", or bass pattern. That would not only include lute player Thomas Mace, teaching students in Oxford, but also a country fiddler in Derbyshire doing "divisions" on the theme of a 3/2 hornpipe, or blind Jacob van Eyck playing variations on popular song melodies with his recorder in a bell tower above Utrecht. However, even then, those variations would often be written out for musicians to learn. The appeal is obvious. Why come up with your own, when somebody like van Eyck has written out some good ones? And, not everyone is going to be talented at improvisation. That continued into the 18th c.

But with the advent of large ensembles and more complex compositions, less freedom was there for the individual musicians to do what they wanted, and composers specified more exactly what they wanted them to play. Bach may have been able to use a figured bass notation that, like a lead sheet for a jazz musician today, let the accompanist know what general harmonies were expected. But Mozart wanted much more control over all the lines in a score, and Beethoven far more than that. There was also just the essential simplicity of theme-and-variation, and also the composer/performer ( like Chopin, or Schubert) became less common, and as the professional composer began to predominate, there was less room for the inclinations of the performer. Especially an ambitious composer, like Beethoven who wanted to do more complex harmonies. Beethoven would write variations on themes, in his lighter stuff, but considered his lighter stuff a way to pay him to be able to write his heavy stuff. The form got less respect. Late examples of theme-and-variation can be kind of comic, like Charles Ives' variations on America the Beautiful.

And where the composer left off, after writing out the specifics in the score, the conductor would take over, specifying the interpretation of it he/she wanted to hear. Originality on the part of the musicians was not something that often sat well with either composer or conductor. Still, some more modern composers tried to allow choices- like George Crumb, allowing performers to choose alternative sections to play. Not quite improvisation, but an acknowledgement that maybe the performer had some talent.

However, one place where improvisation continued to be used quite late was in cadenzas. When the soloist for a concerto came to the end of a movement, he/she was expected to be able to improvise something to cap things off. That could be annoying at times- Haendel once conducted a concerto with a violinist named Dubourg. At the cadenza Dubourg wandered fairly far, and the orchestra had to silently wait for him. As he finally reached the last dominant, Haendel was heard to call out "welcome home, Mr. Dubourg!!".

Improvising a cadenza was common, well into the 20th century. For those who didn't want to do that, plenty of cadenzas had been written out by then, but it was still not unknown. And it seems to have returned- plenty of soloists now will come up with their own cadenzas. And of course in jazz improvisation has never gone away, and plenty of classical performers, working in small ensembles, have found improvisation to be something that adds some spontaneity to the music.