How did jazz become an upper class pursuit when it was considered uncouth and unrefined fifty years ago?

Difficult to answer the question as phrased, because there are many preconceptions attached to it. But the most glaringly obvious fact is that jazz was created, perfected, and expanded by African-Americans, who were subjugated by apartheid-like Jim Crow laws that were in effect until 1965. If jazz was considered "uncouth and unrefined", it was only racial prejudice.

In the period roughly 1920-1970, it was possible (though never easy) for jazz musicians to earn a living by performing, recording, composing. Since then, jazz has lost popularity, and has not generated enough economic activity to sustain its practitioners.

Jazz was formerly taught through informal apprenticeships. Now it is taught in universities. A high percentage of current performers are graduates of these universities.

There is also the question of the changing demographics of the audience for jazz. At some point in the 1960s or 70s, African-American audiences abandoned jazz and embraced soul, funk, R&B, and later, hip-hop as the voice of their community. It's a chicken-and-egg problem, but at the same time, jazz performers began thinking of their music as a form of fine art, not a popular idiom, and their innovative and often radical expressions no longer reached a popular audience. The audience for these uncompromisingly artistic forms tends to be white, educated, and not poor.

Also, throughout the history of jazz, Europe has served as a haven for jazz musicians who found a more supportive environment to live and create in. Since European governments provide more funding for cultural innovation than the US, there has been a tremendous upswing since the 1970s in the number and quality of home-grown European jazz musicians, who are, obviously, white.

Sources: Amiri Baraka: Black Music
Edward O. Bland: The Cry of Jazz (film)
Ekkehard Jost: Free Jazz
Stanley Crouch: Kansas City Lightning