The distinction here, is speed. It is the most important factor to the rate at which a given horse fatigues. This shouldn't come as a surprise; after all, an intense sprint takes much more energy than an easy jog and it's only little different for horses. Unlike the general condition of the horse or the weight it carries^1, the speed has a roughly logarithmic relation to the time and distance the horse can last.^2 Without delving too deep into the math and the approximate formulas that can be made for how long horses can run^3 , the most important take from this is that a loaded horse can comfortably trot for hours, but only gallop/charge for a couple minutes at best. The reason your polo player switches horses is because it's the logical choice to use the horse at a high intensity. You need it running at top speed to win the game. Imagine for a second that football ^(the non-american one) allowed you to unlimited rotation of the players in the field every 5 minutes, and every top level team had hundreds of players in reserve. The players would likely go from the moderate-intensity running they do now, to just constant sprinting across the field, to get all the speed advantage they can get ^(sounds fun actually, I'd love to play a match like that).

But back to the horses and battlefields: the simple answer then is that cavalry wasn't constantly charging/galloping, and fatigue management was as much a concern for cavalry as it was for infantry units. Both could trade speed for endurance or vice versa, which makes for crucial choices in warfare of any period, as good maneuvering alone could already win you entire battles. An enemy on a hill is a textbook case where the 'wrong' approach would be to charge troops (infantry or cavalry) up the hill and then fight. The added exertion and lack of rest gives a decisive advantage to the defender on the hill. This type of fatigue (e.a. "being winded") for both humans and horses is of shorter duration than one might think, and a unit that is given a short pause may easily find itself back up to near-normal fighting strength, but nonetheless it stands to reason that both infantry and cavalry didn't sprint without good reason and would rather pace themselves. Cavalry, it should be noted, has a far higher capacity to cover intermediate distances at a high pace than humans, giving them a distinct advantage in maneuver at the scale of minutes to single digit hours ^4; a heavily loaded horse can still cover up to 2-3 times as much distance as a human in one hour.

The high-intensity part of battle is really only the fighting, and the charge (if at all present). The intensity of sprinting, galloping and melee fighting is so high that it is physically impossible for soldiers, be it on horseback or on foot, to perform these actions for more than a couple minutes. It follows logically that, when analyzing sources that speak of a 'battle' taking hours or days even, we should always consider carefully what the source means with 'battle'. The mainstream idea of a battle is just the fighting, but more often than not the maneuver and skirmishing that precedes it, and the chase that follows, takes up the super-majority of the time. So to sum it up: a heavy cavalry unit might spend the first couple hours of a battle just casually trotting around (or even standing still), and might occasionally increase its speed to secure a certain position on the field if that's warranted, but it'll reserve its high-intensity sprint for when it's absolutely necessary, such as a charge, or a quick flanking action, and this only takes minutes at most, covering rather modest distances.

I really couldn't sleep over this thought :)

And I wouldn't be able to sleep tonight if I didn't workout this question, so I'm glad to deliver!

---

1. The Cataphracts and medieval knights used very heavy breeds of war horses to carry the load. This is far from universal. Plenty of cavalry throughout the ages was far lighter in armor and armament than these juggernauts, meant to harass lighter troops, skirmish, scout and give chase. These likely emphasized speed and endurance more and would outrun their heavy counterparts, which had to ration their energies more carefully. Similarly, not all cavalry were... elite. Smaller, cheaper horses, down to mules and outright shitty horses have all been used for battle. The load and quality of the horse obviously matters a lot to how long it can run at a certain speed, but the effect is not nearly as pronounced and your question was mostly about the heavy cavalry.

2. "An Approximate Law of Fatigue in the Speeds of Racing Animals", A. E. Kennelly, Proceedings of the American Academy of Arts and Sciences, Vol. 42, No. 15 (Dec., 1906), pp. 275-331

3. Since you probably didn't come here for math (I'm gonna get cursed at if I include too much math on this sub), and it strongly depends on the individual horse and situation anyway. The paper I referenced analyzes record-holding racing horses in the late 19th century, which is a very far fetch from war horses in ancient/medieval times loaded with armor.

4. Interestingly enough, at very long distances (>20 km), humans beat horses again, because we're really good at endurance running.