Big cities being mainly German-speaking (or Polish/Italian in the cases of Lviv and Trieste) regardless of the language spoken in the surrounding countryside seems to have been the rule rather than the exception until the middle of the 19th century. How did this come to happen?
How did Budapest, for instance, manage to retain a German-speaking majority throughout hundreds of years of being surrounded by vast numbers of Hungarian speakers in the countryside? Were Hungarian speakers disallowed from moving to the city, or did they have to start speaking German if they moved to Budapest in order to get a job or do business?
And how did this come to be in the first place? Did the natives of Budapest and other cities slowly start speaking German over the centuries due to influence from German Austria, or did German-speaking Austrians move to Budapest and displaced the Hungarian speakers? (And in that case, why did they leave?)
I can't speak specifically for Austria-Hungary, but a similar phenomenon existed in other cities throughout Central and Eastern Europe. I suspect there are similar motives for both regions.
Remember that until only a few decades ago the vast majority of the world's population lived outside cities in rural settings (mainly farms and the like). There was a clear divide between the urban and rural populations, and very little movement between the two (urban folk had no reason, or skills, to take up farming while the rural class rarely had the means to make the move, or any skills either).
It also didn't help that there was ethnic differences between the two: the rural class was often the dominant group, while the cities were comprised of a select few (Jews and Armenians being a couple prominent examples, the latter especially in Ukraine). As such they would speak their native language, while the rural folk kept to their own. Over time this led to more division, and the local language (Ukrainian, Belarusian, being major ones in this region) seen as a "lower" or "peasant" dialect for simple people, while anyone educated or prominent would speak a different language (Polish, Lithuanian, German; Yiddish to some degree). Anyone who wanted to move on from the rural life realised they couldn't speak their native "farmer" tongue and would need to learn the more "advanced" one. It was only with the rise of industrialisation and the mass migration into the cities (which only occurred in Ukraine in the early 20th century) that the local language began to be used in large number throughout the cities.
It's also worth noting that this division was a major factor in the rise of nationalism in the region, and the creation of a separate Ukrainian and Belarusian identity can be tied to this (prominent nationalist leaders of these groups tried to promote the idea that their languages weren't "peasant tongues" but could be used in a more advanced setting).