If the ladder base was far from the wall the defender's stick wouldn't be long enough to push it over, ladders would be pretty heavy with a few people on it, and a defender would be exposed to attack from below. This thread goes a bit more in depth, but these seem to be the main reasons

Assaulting with siege ladders wouldnt be undertaken until the battlemet had been partially cleared of attackers through catapult/trebuchet/arrows. However even if the battlemet was full of men the attackers could easily overwhelm them if enough ladders were used, the first attackers on the ladder would have a rough time but the following waves would break through and create a lodgement. Also siege ladders eventually gave way to siege towers in cases where the defenders could mount a solid defence from the outer wall. With a siege tower all the attackers would be commited at once and success rate was exponentially higher

hooked tops on the ladders made it harder to do this as well