Horses are weak and vulnerable?!?! Are you thinking of the ponies at the petting zoo?

The charge at the gallop is the basis of all modern cavalry attacks (post-1650) and it is well-attested. Whether or not it looks to you like it would fail, it seems to have worked for hundreds of years.

That's because the "average horse" weighs 900 lb/408 kg, and stands around 58"/1.47 m at the shoulder -- or larger. Charging at the gallop, a horse is scary in a very visceral way, especially when there's a hundred of them coming down on you.

To this day, many metropolitan police departments have mounted units for crowd control, because little puny humans don't want to crowd great big beasts that both bite and strike, and can severely damage them. A horse can grab a man in its teeth and pick him off the ground to try and shake a piece loose. Talk to horse people about how you never completely trust them, especially not stallions. "It's always those sweet gentle old bastards that kill you," to quote a friend. (All medieval European warhorses were stallions, whether destriers, ordinary chargers for the men at arms, or rounceys. It's established in the books on horsemanship, backed by lists of horses for "insurance" purposes: if the horse you take to the war gets killed, the crown re-imburses you, and they are all one sex. Ayton, Andrew; Knights and Warhorses, Military Service and the English Aristocracy under Edward III; Woodbridge, UK, The Boydell Press, 1994)

Also, for the person with the sword to put it through the horse would require it be riderless, and probably blind. The rider with a sword or lance is a much greater danger to the person on the ground, and is presumably smart enough to guide the horse out of danger.

Now, Hans Delbrück, in his History of the Art of War, does argue against a modern long charge at the gallop for the medieval line. As the examples he marshals all value the line making contact as a line, not a ragged scatter, a number of period leaders demand a slower approach ride to bring the lines within range of each other. Remember, this was not cavalry charging a line of infantry: it was two lines of mounted warriors crashing together. But it must never be forgotten that mounted charges are as effective as they are fast: if you are only at a trot, you have much less impact, and will be overborne by those who have more velocity. That's basic jousting.

If anything, the slower the line, the more vulnerable to infantry.

Sources:

Delbrück, Hans. History of the Art of War: Medieval Warfare, v. 3; 1929, Berlin; trans. Westport, CN, Greenwood Press, 1982; trans. by Walter J. Renfroe, Jr., Brig. Gen. US Army, Ret.

Hall, Bert S. Weapons & Warfare in Renaissance Europe; Baltimore, Maryland, The Johns Hopkins University Press, 1997

Hyland, Ann. The Medieval Warhorse From Byzantium to the Crusades; Conshohoken, Pennsylvania, Combined Books, Inc., 1996; originally London, Sutton Publishing, Ltd., 1994

Wise, Terrence. Medieval Warfare; New York, Hastings House, Publishers, Inc., 1976; Nice, if broad, discussion of medieval tactics and the effect of changing weapons, also good overviews of individual weapons and how they were used. Good for understanding the threat to horse and rider on the medieval battlefield.