Of course, during this period jazz was commonly associated with young men and women, flappers and the like. However, was there a significant amount of jazz fans of a more elderly age, such as 40 onward?
I don't have any sources on elderly early-jazz fans, so let me look into that!
What I do know is that jazz was originally functional music, this was music meant for dancing, for entertainment in bars (and occasionally brothels!), and for special events. So I wouldn't characterize it as predominantly a youth activity. Early jazz would have been more easily understood as rooted in certain races and classes, with less of an age difference.
Certainly by the early 1920's we find the music in a fad-like state, with wide appeal to a large portion of the American (and French) public. However, your question isn't whether it WAS youth-based, but whether it was SEEN as youth based, and there is a dissonance between how cultural movements operate and how they are seen to operate. With the emphasis on dancing and "flapper culture" and all that jazz, there was certainly a feeling that this was young people's music (even if our records show that many older folks were actually present for the music as well).