What outcry we do have is from Christians. The best example is Tertullian who thought that the games were terrible wastes of time and sinful entertainment. His distaste is founded on Christian morals and thus had little influence outside members of his own community. This seems like a reasonable outcry, although they would probably be more like PETA than the ASPCA in terms of their reception and impact. This was however, only an ideal and many Christians still went to the games even after the religion was legalized.

Outside of that community there were some Roman thinkers like Seneca who found the killing of prisoners distasteful and Cicero who thought the animal hunts were actually pretty boring. They also felt like a gentleman should never get too involved in the displays.

So yes there were people who disliked the games, but there is no evidence that they actually tried to have the games ended until Christianity was made the official religion. Even at that point the moralizers we can still read expect that games will continue but that Christians should not go.

Here is the work by Tertullian I was referencing above. It is pretty manageable if you want to see some of the objections he had to games.

As qsertorius points out, the main opposition came from the Christians. However, the Christians weren't the only opponents to the games. Many Roman Philhellenes looked down on the games, and believed that they were barbaric.

Cicero was one notable example. He stated in the Tusculan Disputations (2.17.41):

The gladiatorial spectacle is wont to be regarded by some as cruel and inhuman, and I know not whether, as it is now managed, it may not be so. But when criminals fought in the arena, if there may have been for the ear, there was not for the eye, any stronger discipline for the endurance of pain and death.

Cicero (106-43 BC) was notable for scaling back the games he offered as aedile (it was customary for the aedile to offer lavish gladiatorial games), and instead, he appeased the populace by offering meat from Sicily and Greek theater performances.

Seneca (4 BC-65 AD) was another critic of the games. In his Letter 7 (7.2-5), he states:

But nothing is so damaging to good character as the habit of lounging at the games; for then it is that vice steals subtly upon one through the avenue of pleasure. What do you think I mean? I mean that I come home more greedy, more ambitious, more voluptuous, and even more cruel and inhuman, because I have been among human beings. By chance I attended a mid-day exhibition, expecting some fun, wit, and relaxation, – an exhibition at which men's eyes have respite from the slaughter of their fellow-men. But it was quite the reverse. The previous combats were the essence of compassion; but now all the trifling is put aside and it is pure murder. The men have no defensive armour. They are exposed to blows at all points, and no one ever strikes in vain. Many persons prefer this programme to the usual pairs and to the bouts "by request." Of course they do; there is no helmet or shield to deflect the weapon. What is the need of defensive armour, or of skill? All these mean delaying death. In the morning they throw men to the lions and the bears; at noon, they throw them to the spectators. The spectators demand that the slayer shall face the man who is to slay him in his turn; and they always reserve the latest conqueror for another butchering. The outcome of every fight is death, and the means are fire and sword. This sort of thing goes on while the arena is empty. You may retort: "But he was a highway robber; he killed a man!" And what of it? Granted that, as a murderer, he deserved this punishment, what crime have you committed, poor fellow, that you should deserve to sit and see this show? In the morning they cried "Kill him! Lash him! Burn him! Why does he meet the sword in so cowardly a way? Why does he strike so feebly? Why doesn't he die game? Whip him to meet his wounds! Let them receive blow for blow, with chests bare and exposed to the stroke!" And when the games stop for the intermission, they announce: "A little throatcutting in the meantime, so that there may still be something going on!"

Note how each man criticizes the games of his era, while saying that the games of the past were more honorable. This is particularly interesting, given that Cicero lived 3 generations before Seneca, and Seneca complains that the games were used as punishment of criminals, while Cicero complains that they were not.

It wasn't always only the Roman thinkers/elite or Christians who showed displeasure at the games. Cicero talks about a time when Pompey displayed elephants against criminals and apparently the crowd found the sounds of the elephants in fear so distressing that they jeered Pompey.

Might be more of a testament to the apparently lasting tendency of humans to care more about animals in distress than other humans though - and clearly it was only a fleeting occasion.