I read that it took a lot of selective breeding to get horses to the level of strength of being able to carry mounted men, which is why bronze age used chariots exclusively.
Is there any fossils to compare directly how large they were, compared to modern horse breeds?
While it is true that horses were of 'smaller' size than modern horses today, to state that it was the reason for the domination of chariots has been shown to be misleading. Height alone does not have significant influence on weight carrying ability as the way the horse is built can compensate. Vets today may recommend that a horse carry no more than 15-20% of its weight, but this is not set in stone. For example, icelandic horses on average stand between 13-14 hands and weigh around 800 lbs, yet the breed is well known for its weight bearing ability. Additionally, people back then were also shorter and smaller than the average man today, but like horses, not significantly smaller. What follows is going to focus primarily on equids in the ancient near east.
There is evidence that suggests that riding on horseback wasn't uncommon even in the days of chariotry. This artifact, from between 2000 BCE to 1800 BCE quite clearly suggests that a man is riding an animal, yet the dating puts it clearly in the time of the chariots. Questions can certainly be raised which animal he is riding as humans had also been using onagers, donkeys, and cross breeds (such as mules) during this time period as well, but riding did exist. When horses became the preferred war mount is hard to say, but their size appeal is quite apparent when compared to the other equids (which were also never phased out completely anyway). Donkeys, on average, stand at around 12 hands or less. Onagers range from less than 10 to 12 hands depending on the species.
Horses during this time period have been found to be comparable to modern saddle horses between 14 and 15 hands. On the high end, a specimen from a Hittite burial site near Hattusas in Anatolia was estimated to stand at 15 hands. These horses were clearly more than capable of carrying riders and their equipment as we have depictions of horses and riders from all over the age of chariots, both in time and space. Plenty of archaeological evidence has also been discovered ranging from Egypt to Turkey. Since you mention comparing to modern breeds, the following may be useful.
| Breed | Average Height in hands (measured to a horse's withers) |
|---|---|
| Clydesdales | 16.1 - 18 |
| Shire | 16.1 - 17.3 |
| Thoroughbred | 15.2 - 17 |
| Lipizzaner | 15 - 16.1 |
| Akhai-Teke | 14.2 - 16 |
| Andalusian | 15.1 - 15.3 |
| Arabian | 14 - 15 |
| American Mustang | 13.2 - 15 |
| Haflinger | 12.3 - 14 |
| Welsh Pony | < 12.2 |
| Shetland Pony | < 10.2 |
As you can see, this is not a significant difference in size from modern horses. Clydesdales and almost all draft breeds will be significantly larger and heavier, but the vast majority of the draft breeds have come about within the last 200 years, too late to have been bred for knights and armor. Most other breeds like thoroughbreds will not be significantly larger. On the low end they are barely larger than the remains that we have found.
There is no question that horses have been bred for specific traits and certain breeds have been bred larger (while others have been bred smaller), but to say that we couldn't ride them until after we selectively bred them is unfortunately not supported by the evidence uncovered of the past. Humans were clearly riding equids despite the dominance of chariots; and at up to 15 hands, horses really weren't significantly smaller nor were they incapable of bearing men and armor.
Sources:
The Horse in the Ancient World by Ann Hyland
International Encyclopedia of Horse Breeds by Bonnie Hendricks