It seems like if a unit of horsemen even heavily armored would be let down by the fact that horses are really the weak spot as far as having effective damage done to an individual. What I mean is that it seems like any wound landed on a horse would freak the animal out making it hard to effectively control. Also when a pack of horsemen charging at galloping speeds hit a organized unit of infantry, sure it would hurt alot of men on the ground but it seems like the horses would just trip over eachother, the men they hit, shields, rocks, etc and fall and just generally hold up the rear ranks of charging cavalry. Are horses just more agile than I am imagining or was the nature of combat just not what I'm imagining?
First I have to ask you; are you aware how large a typical war horse was? A good Destrier could average and even exceed 15 hands, other breeds similar; making an average warhorse or charger over a meter and a half in size. Asides from being incredibly close to the height of (for a good chunk of human history) an above average man, it suggests quite a bit of muscle mass.
Secondly I posit this to you; particular breeds of horse were bred for aggression and unruliness. Finally, bear in mind that horses, like any animal, can be socialized to certain situations, even one as egregious and harrowing as combat. In short, chargers and Destriers had the perfect temperament and body-type for hazardous environments.
So this is not necessarily true:
What I mean is that it seems like any wound landed on a horse would freak the animal out making it hard to effectively control.
It would take a great deal of trauma, and perhaps the death of its rider (i.e: no one to reign the creature in), to cause a Destrier or other war-like breed of horse to panic to the point of being uncontrollable, if it was properly raised, broke and socialized. This is of course discounting more material accessories to aid in keeping the Horse calm and under control, such as blinders.
Also when a pack of horsemen charging at galloping speeds hit a organized unit of infantry, sure it would hurt alot of men on the ground but it seems like the horses would just trip over eachother [sic]
Well, sure, if it was a disorganized rabble. A charge is rarely 'hell for leather' - it certainly isn't if it wishes to be effective. A charge begins at a trot, and slowly evolves from trot to a short gallop before the moment of impact. The reasons for this are manifold but I'll touch on a few relevant ones: (1) preventing the horses from 'blowing' their charge with wasteful movement, and (2) keeping the formation from being 'strung out' and in good order (addressing your issue, I believe). It is worth noting that a charge against still disciplined infantry is high risk and was rarely done as well. Heavy Cavalry break formations, true, but they have to be breakable to begin with; tired, panicking, strung or thinned out, etc.
Are horses just more agile
Horses are incredibly agile creatures, and they are far from ungainly or awkward in gait. They have consistently shown their strength and intelligence, and their ability to bear great weight and retain the ability to put on great bursts of speed, if even for a short time.
I recommend, especially for the Napoleonic period, reading books by Dodge, Chandler and Elting's Swords Around a Throne; all of which go out of their way to explain the intricacies an discipline necessary for a coherent charge, and how the right breed and training of the mount can make all the difference. Chandler even goes out of his way to say that French horses and horsemen were rather lackluster, but the training and coherency for both rider and mount gave them quite a presence on the battlefield despite the lack of equine skill on part of the rider. He likewise heaps praise on Russian and British cavalry, which was often as well disciplined and as skilled if not more so than their counterparts.
/u/BritainOpPlsNerf has provided an excellent general overview; allow me to cite a specific case to demonstrate how a cavalry charge could be conducted without confusion and disorganization.
The 11th-12th century Norman milites, usually translated to "knights," fought in tactical and administrative units known as conrois. The conroi consisted of anywhere from ten to thirty knights, often members of the same lord's household, or hailing from the same locality; in any case, they knew each other well, had trained together extensively, and were capable of operating as a coordinated unit, in conjunction with other conrois. Depending on their number, they would array themselves in two or three widely spaced ranks. By keeping distance between themselves, they could conduct very sophisticated and flexible maneuvers.
The first rank might either charge home or break away at the moment of contact. If it was the former, it could either peel away after delivering the charge to enable the second rank to press home, or the first rank could remain in the melee and the second rank could surge through the gaps and exploit the situation further.
If the defending infantry was unbroken and judged too strong to be taken by a direct charge, however, the first rank might break away, usually after delivering a pelting barrage of thrown spears. It could then retire through the second rank, while the second rank, and then the third, repeated the maneuver.
Yet another way, one well attested to by not only the Normans but Bretons and other northern French horsemen, was the feigned retreat. This is exactly what it sounds like: an attack made with great sound and fury, but which broke away at the last moment and retired in mock disarray. The goal was to lure the defending infantry out of their shieldwall, and then swiftly turn and cut them down.
All three could be used in a given battle. If we take Hastings, for instance, the Norman knights seem to have begun the battle with volleys of thrown spears. They then seem to have made a feigned retreat, or perhaps a series of feigned retreats, which succeeded in drawing out numbers of the Saxon infantry. Once the Saxon line had been severely weakened, they then pressed home in a frontal charge that appears to have broken the Saxon shieldwall and driven them from the field.
A simple answer to your question is that the nature of combat is not what you are imagining. Cavalry very rarely charged into heavy infantry formations, and were very rarely successful if they did. As has been pointed out elsewhere, even when we are talking about armoured knights, the apogee of shock cavalry, they would only charge home if the foot formation that they were charging wavered. A horse, no matter how well trained, is reluctant to run into what looks like a firm obstacle. It will seek to jump over it, or will look for a gap to go through. If it sees neither it will be likely to stop rather than run into what it knows is likely to cause it pain. A formed steady mass of infantry will appear to the horse to be an obstacle too high to jump and too solid to go through, so the horse will be likely to stop. The descriptions from Napoleonic battles, such as Waterloo, of cavalry charging up to a square and then either stopping or going around the square typify the standard reaction of a charging horse to formed infantry. If, however, the infantry weren't formed or weren't steady and were wavering, the horse would not perceive the infantry as a solid obstacle, they would see gaps through which they could continue, so would neither stop nor swerve, but would charge home. The wavering or unsteady or unformed infantry would then be shattered by the horses passing through the gaps, shouldering men aside, and by the weapons being used by the cavalry, be they lances or swords. Weapons and formations used by infantry to repel cavalry, phalanxes, squares, pikes, long spears, bayonets, were essentially morale boosters rather than physically effective weapons. If the formed infantry were confident that their spear or bayonet could keep off attacking cavalry they would confidently stand their ground and the charging cavalry would probably not charge home. There are, of course, exceptions. Squares were broken in Napoleonic times, for example, but so rarely that the occasions where this happened became famous, and have created an erroneous impression that squares were not that effective. However, in nearly all cases where a square was broken there was a specific cause of this happening. One of the most famous was the KGL dragoons breaking three French squares at Garcia Hernandez. As they charged a mortally wounded horse fell next to the French front rank, it's death throes causing it's legs to lash out. A quick thinking officer led his men into the gap caused by the French infantry getting out of the way of the dying horse, others followed and the morale of the French infantry crashed as they realised that the wall of the square had been broken and they broke and ran, or surrendered on the spot. Two other squares seeing the first square became unsteady and broke in turn as the charge swept up to them.