Two big reasons.

I'm not sure if you've ever been around an angry horse, but they are quite a sight. A horse in battle isn't just standing and maneuvering about with consideration for it's surroundings, it's 500kg of muscle and hooves that is highly stressed and trained to attack with kicks. They are also twisting about, and are huge. Horses are fast, and very strong. A single kick could easily kill a man armor or not, and a horse will often fight on despite serious injuries. Now add to that an armored professional soldier who is swinging a weapon at you with the advantage of higher ground over you. Your swings up will be weaker and less accurate while his swings down are stronger.

The second reason is that cavalry's strength is in it's maneuverability and quick impact or shock tactics. Generally cavalry would ride in and attack, then likely either break away to regroup and attack again or pursue a fleeing enemy. They wanted to quickly break a unit and pursue it to ride it down, not stand in a protracted melee. Cavalry never wanted to be stationary because what you said could and did happen.

The best way to counter a cavalry charge was to stand your ground in a tight formation so that the cavalry has to halt or check it's charge and engage you without all the shock and impact. If this happened cavalry would often break off from the fight and try again. Cavalry was able to do this because they could get away from the melee quickly and would not be run down or caught in turn. It took an incredible amount of training, bravery, and experience to stand in the face of a cavalry charge but a veteran would know it was your best chance to survive.

Edit: Finding any scholarly analysis on this was harder than I thought! I had been taking a professors word on it but I did manage to dig a few things up. As one might expect training manuals from ancient times are hard to come by.

Patton wrote a manual on Sabre use, and deals with horseback quite a bit. The book, more of a pocket reader, is called Manual for Sabre Exercise. Patton describes the rider of the horse as a second weapon in the whole package of mounted combat, and emphasized that one should ride through and over opponents, not past them.

The saber is solely a weapon of offense and is used in conjunction with the other offensive weapon, the horse.

He is a unit in a line rushing on the enemy with the one idea of riding him down and transfixing him with his rigid saber, held at the position of charge saber.

Patton wrote three or four other manuals on cavalry and sabre usage, most of which were out of date within a few years outside using them in colonial areas without modern weaponry to oppose you.

Moving backwards in time there are writings regarding cavalry charges into infantry squares (generally a bad idea) regalling us with tales of horses charging full speed and jumping into and over the lines of infantry. I can tell you from working with horses that a horse would not do this without proper training. John Grant Malcolmson won a Victoria Cross for this very act in 1857 against Persians.

Going back more to the middle ages, The Schola Forum (a source of much of these sources) found some interesting images of warhorses being trained to rear, kick, and bite when presented with an armed soldier in The romance of Alexander. Link 1 Link 2 Link 3

I can't find them off hand right now, but warhorses from that time are well noted for being mean and aggressive, often biting and kicking grooms. It may be that aggressive horses were well suited for battle and that kicking and biting was simply not discouraged; however, it appears they were trained to specifically attack armed soldiers in battle.

Edit 2: I didn't even get into modern crowd control use of horses which uses a wall of horse bodies to direct crowds, or horses in Polo which can be quite violent.

Cavalry charges were powerful against broken infantry lines and were often used to break a formation. The cavalry needed to be disciplined, in formation, highly skilled, and outfitted with the best arms and armor for a charge to be effective. If an infantry line did not rout after a charge and the cavalry became entangled with their enemy the cavalries effectiveness decreased dramatically and the infantry attained the advantage. The smart move for cavalry after a failed charge would be to disengage and prepare for another attack. Also steel armor wasn't particularly heavy and a knight could easily get up after falling from his horse, so long as there weren't enemies around him keeping him down.