Just how fast was one of these things? I've heard Sherlock Holmes used one because of how fast they were.
If you compare a Hansom cab to a horse-drawn wagon, you'll notice that it's shorter. Think about that wagon going around a corner: the horse has to actually walk straight forward and pull the wagon past the corner, then pivot for it to make the turn. If it doesn't do that, and pivots instead right at the corner, it drags the wagon against the corner of the building...or drags the wagon onto the sidewalk into the pedestrians. This could be damaging- corners of buildings, entrances of driveways, would not uncommonly have guard stones set into them because a heavy wagon could actually dislodge stones and bricks if dragged into a corner, and that would substantially weaken the building. And of course it's also not pleasant to have a wagon drive over you, if you're a pedestrian. There's also the problem of moving traffic. The horse has to actually step forward into the cross street: have you ever been driving down a road and had a long truck emerge from a cross-street, the driver having to swing into your lane in order to make the turn? Imagine this happening constantly in the middle of a crowded city. Like that truck driver, the wagon drivers would have to wait for a chance to pull forward and block traffic, or would try to bully their way into the cross-street. In either case, the traffic would slow. The shorter a vehicle, the less trouble it would have in these ways getting around in traffic.
Then there was also the problem of deliveries. Like today, commercial vehicles would tend to simply stop in the street to unload . Other vehicles would then have to get around them. If they were longer vehicles, that would also be problem.
A wagon driver would also have to be looking behind, thinking about how his wagon was turning, as he also watched his horse in front. The Hansom cab driver would be sitting behind the wheels, and so could clearly see both where the cab was going and what the horse was doing.
When Holmes/ Conan Doyle said a Hansom cab was faster, he might have been thinking about how it was light and could be pulled by just one horse at a fast trot, and how the big wheels could hop over small obstacles. But he would very much be thinking of how the short Hansom cab could more quickly maneuver in traffic.
For an idea about how it might have been , here's a short film of a somewhat wide street in London from the period. Imagine the mix of vehicles and pedestrians happening on a smaller street.