Hi there! There were various sports in the ancient world that probably would have resulted in a great deal of physical injury and exertion. Most notable among them would have been boxing, which was popular in both Greece and Rome; there were no weight classes either, so you could be a wheedly fellow pitched against an enormous Goliath and you'd just have to deal with it. They would have bound their hands in leather thongs (ίμάντες) to protect themselves, and sometimes padded gloves as well, though the face would have been the primary target, not necessarily the body. The Romans were even known to use iron weights (caesti) in their gloves (yikes!).

There were also a variety of ball-based games, not unlike American football or even Rugby, such as harpastum in Rome and episkyros (ἐπίσκυρος) in Greece. Galen recommended harpastum as a way of keeping one's body fit, though it was also noted by Athenaeus (c. AD 200) in his only extant work "The Learned Banquet" that harpastum could be quite violent at times -- as happens in rugby and American football as well.

I'm just going to add slightly to /u/RomanImp's excellent response, since he hasn't mentioned the circuses. By the end of the Roman period, when the gladiatorial games were in decline, the circuses were by far the most deadly sport in Rome, and even during the Principate you actually had a higher chance of getting hurt if you were a charioteer than if you were a gladiator. Gladiators rarely fought to actually hurt each other--the aim was generally to disarm or knock over your opponent. But in the circus, in keeping with the traditions of Greek chariot-racing, you were using a light chariot yoked to four horses when it really only had room for two. You had a nice long whip, you were unstable, your opponents were unstable. And all of you are riding at very high speed towards the same narrow turning-post, trying to get as close as possible to it. Accidental crashes occurred in nearly every race, and since it was difficult to spot foul play we have evidence that opposing charioteers would often try to whip each other or force collisions.

Another thing is that there are actually other ways in which boxing doesn't resemble our own. In addition to the leather hand-bindings and the lack of weight classifications, there wasn't any maneuvering. The way Greek boxing (which the Romans adapted for their own purposes) was that it was entirely a trial of determination and strength. So there wasn't a nice big ring to fight in, and the fighters didn't bounce and hop around in the way that Ali and some other fighters have become famous. There was a very small ring, large enough for two men to fit within reach of each other but small enough that if someone took a step back he'd be out of the ring. In the earliest and purest forms there was no ring, but it seems to have been introduced at Olympia sometime early on (Homer's boxing match gives no evidence of a ring). And the two fighters just stood there opposite each other in the ring (in the earliest games it seems that they actually sat opposite each other on some kind of stool) and just traded blows until somebody gave up, fell over, or died. It seems blocking was more or less prohibited, although dodging was fine if you could stay in the ring, and body blows were uncommon, with pretty much all strikes being directed to the head.

Another thing is that the modern conception of the gladiatorial games encompasses several different events that resembled the gladiatorial games but which were not a part of them, and which were far bloodier than the gladiatorial combats. The most important of these, which was considered a real sport, was the wild-beast hunting. These bestiarii would battle wild animals and were considered quite separate from the gladiators--wild animals and gladiators never mixed--and unlike the gladiators, who belonged to a series of guilds and weren't interested in actually hurting each other, the bestiarii were fighting wild animals that were trying to protect their own lives. An awful lot of bestiarii paid for it.