It was difficult to take walled cities and castles through assault by ladders.

The assault on Jerusalem in the First Crusade in 1099 is an example.

About 13,500 crusaders eventually reached Jerusalem on 7 June 1099. The city was well prepared for a siege. The Crusaders were not, having few supplies. The Crusaders knew they could not mount a long siege. On 13 June they tried to storm the city with ladders. This was a total failure.

fortunately, for the Crusaders, two Genoese ships arrived in Jaffa soon after the first assault failed, with some supplies. Unfortunately, the Crusaders learned that a relief army was marching to the aid of Jerusalem from Egypt.

Foraging for wood, (which had been destroyed in the vicinity of Jerusalem by the Fatimid defenders) and also breaking up the two Genoese ships for their wood, the Crusaders built two siege towers to attempt to storm the walls of the city.

By the night of the 14th of July, the Crusaders had built two siege towers, and used them to assault Jerusalem. The south siege tower was set alight by the fire arrows and pots of oil of the defenders, and destroyed. The northwest tower, however, was successfully advanced to the walls. The crusaders fanatically assaulted the walls from this siege tower broke through with sheer fury and determination and opened a gate. Thus Jerusalem was taken.

In this case ladders proved unsuccessful in assaulting a walled city, but one of the two available siege towers managed the job.

http://en.wikipedia.org/wiki/Siege_of_Jerusalem_(1099)fan

Ladders were not always unsuccessful in castle assaults, but they were a very risky and costly method which did not often work. (Siege towers were also risky and often did not work.) Castles were designed to be defensible by small garrisons against large numbers, and were difficult to take by assault.

The reliable way to capture a castle or a walled city in Medieval times was to besiege it until starvation compelled surrender. If this was impractical (as in Jerusalem, where the besiegers would have perished from lack of food and water long before the defenders) an assault could be attempted, but it was usually a risky and costly business.

I agree with the arguments previously stated. Assault on walled fortifications in the archaïc (Greek) period was characterised by prolonged sieges. It was not uncommon for sieges to take more than a year.
As with almost all sieges, decisive victory isn't usually gained from the employment of a single sort of device. This counts firmly against the case of the ladders. In many romanticised depictions the use of ladders is overrated, though not grabbed out of thin air. For the archaïc period some examples are noted in the works of Thucydides on the Peloponnesian War (3.51 and 4.13).
On the other hand ladders do were employed in other cases but the direct assault on a walled city. Take the case of the construction of the Athenian long walls (460 it was?). These walls were meant to encompass a general territory and to make it easier to defend against a grand intrusion. Ladders in this case were employed to sneak in because they are handy, fast, and easy. Ladders then are used in concordance with covert tactics; something the Lacedaemonians (Sparta) could never abhore to, as Herodotus (9.17) aptly states.
Edit: Almost forgot, Nic Fields, Ancient Greek Fortifications (2006)

I cannot speak for medieval sieges but I can enlighten the ancient side of the question. More specifically the romans. In ancient times sieges was something that took considerable effort. To take a large and well fortified city was extremely difficult, and could take years. Ancient sieges tended to consist of a move and counter-move idea as the attacker and defender deployed their engineering skills and massive labour force to gain an advantage and/or negate a project begun by the other side.

Attempt to take a city by storm was something that was very risky. It was certainly likely to involve a high number of casaulties and it was no guarantee that it would succeed, and if the defenders were organized and skillfull one could rarely hold on to the ground one managed to gain. Treachery was something that was relatively normal and the easiest way to capture a city was to gain contact with political forces inside the city who would find ways to help the besiegers. One could also try to starv out the city, but this took a long time and presented the problem of obtaining food and supplies because one was staying in the same place for a long time (especially true in foreign territory).

But to get back to your original question. We can take the Siege of Carthage as an example. The siege lasted for years and the Romans made several unsuccessful attempts to assault the city. Siege engines, siege towers, siege rams were constantly built and burnt down by the defenders.

So in summary I would say the use of ladders and the storming of city was something that was very risky and only done when the general felt he had a good chance to win. It often involved many casaulties, and there was no guarantee one could hold onto the ground one gained in the long run. The Siege of Carthage is a great example of how extremely difficult it was to take a city of that size. The final assault one the city came from the secure base of siegeworks that had been constructed with great effort over a very long time and when the defenders were very weak from starvation. So using ladders and storming a city was not done lightly. At least in ancient times.

Source: The Fall of Carthage and Caesar by Adrian Goldsworthy