In modern warfare, urban battles are very deadly to invaders (see for example, Battle of Hue, inter alia) and seem to be one of the forms of warfare that is most punishing and least appetizing for advanced militaries (the Germans' experience in Stalingrad, the Americans' experience in defeating the insurgency in Iraq, for example in Fallujah).
In the medieval and classical period, it seems that all the focus (and all the difficulty) in conquering a city was based on breaching the city's walls and getting one's soldiers inside. Once the soldiers are in, the city's inhabitants seem to have no ability to defend themselves. This is in contrast to modern warfare where even a small group of resistance fighters can tie down, hinder, and bleed an invading army for days or weeks. What changes account for this shift in dynamics in urban warfare?
There are two key factors:
For the defender: firepower, and firepower that can be effectively used from concealment.
For the attacker: terrain that restricts the use of armour and firepower such as heavy artillery.
The first forces defensive positions to be dealt with before further progress can be made, even if they are defended only by a small force (possibly even 1 fighter). The first problem is that the enemy position must be located, and it might be well-hidden in rubble. The second problem is that either the position must be destroyed, or its abandonment forces. The restrictive terrain means that only a small portion of the attacking force can fire on the defensive position, and it might be impossible to bring heavy weapons to bear on it. If heavy weapons are available, it is often only possible to use them in direct fire, so it can take time to bring them into position. Simply crushing the position with armour is very risky, since the channeling of armour along roads makes armour vulnerable to mines and ambush, and the defensive position might well be able to destroy armour.
In open terrain, once a strongpoint is located, it can often be destroyed or suppressed by heavy artillery (using indirect fire) or air attack. This is not always possible in a city. Rubble, basements, etc. can also give considerable protection against such attack.
Thus, urban fighting can be a slow process of search, locate (often by being fired on by defensive positions), and then destroy the defensive position with a relatively small force. Then onto the next position. If the defender is familiar with the terrain - if they're in their own city - they can often move between defensive positions, and even re-occupy bypassed positions, making the area behind the front line, insofar as there is a front line, dangerous.
The attacker can attempt to speed the process up by the use of heavy firepower - artillery and air attack - whenever possible, which results in widespread destruction in the city (as in Hue in 1969, Grozny in 1999-2000).
It isn't necessary to have urban terrain to achieve these conditions. One example is the Battle of Buna-Gona in WW2 in New Guinea, which was a bloody and grinding two-month offensive against a network of Japanese bunkers and strongpoints, hidden by jungle, and sited for mutual protection. Even if the attacker has plentiful artillery and armour and terrain that allows their use, if these are not used appropriately, they can still be stopped by smaller forces (e.g., the Soviet difficulties with breaching the Finnish Mannerheim line in the Winter War in 1939).
As for the ancient or medieval case, if the defender decided to endure a siege, with the risks of starvation and disease, it was often because they had no realistic chance of defeating the enemy in an open battle. This, if the enemy breaches a wall or gate, they stand little chance in the street fighting. If the numbers are more balanced, things can turn out differently, and the defender can win in the streets, but typically the attacker was more powerful.
In summary, one man with an assault rifle or machine gun hidden at the end of a street can keep the enemy from advancing down the street much more effectively than a man with a spear or a crossbow.