The song is actually by Irving Berlin, not Fred Astaire. Irving first published it in 1929 - Astaire wouldn’t be involved until the 1946 film Blue Skies, which used different lyrics (more on that later).
The original 1929 lyrics (from The Complete Lyrics of Irving Berlin, 2001):
Have you seen the well-to-do / Up on Lennox Avenue / On that famous thoroughfare / With their noses in the air / High hats and colored collars / White spats and fifteen dollars / Spending ev'ry dime / For a wonderful time
If you're blue and you don't know / Where to go to, / Why don't you go where Harlem glitz, / Puttin' on the Ritz!
Spangled gowns / upon the bevy / of high browns / from down the levee, / all misfits, / puttin' on the Ritz!
[…]
Come with me and we’ll attend their jubilee / And see them spend / Their last two bits / Puttin’ on the Ritz
Emphasis mine. And yes, it’s high browns, not highbrows - high browns being an archaic term for lighter skinned black folks.
So, these lyrics are a reference by Irving Berlin to black harlemites having a good time on Lennox avenue. Lennox (also known today as Malcolm X blvd) was the cultural center of the Harlem Renaissance (See: Encyclopedia of the Harlem Renaissance, 2003) that was happening around the same time (more info), and was a popular destination for music, dance, and food. The fifteen dollars isn’t a ton of money - adjusted for inflation, $15 in 1929 is about $250 in today’s spending power. The implication here (with lines like “spending every dime” and “see them spend / their last two bits”) is that these aren’t particularly wealthy black people, but they’re spending all their money to pretend to be.
In more practical terms of what fifteen bucks would actually get you, you can see some outstanding old New York menus here - I highly recommend checking these out. Cheap items on the menu were a dollar or less, but there are lots of fancier options. A filet mignon at the Cotton Club circa 1925 would run $2.25, or a T-bone with French fries at Frank’s circa 1917 would cost $4.50. Much like today, the booze is where it gets expensive - also at Frank's, you could get a bottle of imported wine for $3.50 to $4.50, a pony of Henessy for $0.75, or a pint of Pol Roger Brut Champagne for $6.00. So you can see how you could blow through a whole $15 on a single wild night if you were buying expensive mains and/or lots of drinks!
The 1946 version of the song (for the film Blue Skies - again, see The Complete Lyrics of Irving Berlin, 2001) changed up these lyrics, removing the references to black Harlem (instead using Park Avenue and references to famous Caucasians like the Rockefellers and Gary Cooper) and substituting “lots of dollars”.