Destroying cities, contrary to what you might believe, is surprisingly difficult, and resource-intensive. To this end, cities don't generally get destroyed unless a battle takes place within the city walls, so to speak. I think that Warsaw is a poor comparison to make - Warsaw was place to the famous Warsaw Uprising, which infuriated Hitler who ordered the city to be razed to the ground, explaining the damage left behind, and so to that extent Warsaw was an outlier in the damage that it sustained.

This is not to say that other cities did not suffer either. Due to its seaports and industry, Gdansk (Danzig) got bombed heavily by the Allies, and then a battle ensued within the city borders. Contrary to that, the industry in Krakow itself was not as important, and so the city did not sustain heavy bombing.

Furthermore, Hitler designated certain cities as Festungs, (simply 'fortresses' in German), which were marked as defensive strongholds against Allies. These included Koningsberg, Gdansk, Wroclaw (Breslau), or Warsaw. You will note that all of these cities were heavily damaged during the war, and that is exactly one of the reasons for the damage. Krakow was not designated as such a stronghold, and so the German army was permitted to retreat from the city.

I can't speak for the case of German occupants of Krakow specifically, but retreating from a city if possible is also good idea. The army gets preserved and does not get surrounded, something that is relatively easy when it is contained within a city. Yes, the enemy will take heavy casualties in retaking the city, but it is an all-or-nothing situation.

I'd be happy for someone to expand on this.