I can write about the Late Republic and the Julio-Claudians. I don't know about the later Empire.

I wouldn't say "pacifist" elements, but there were critiques of Roman imperialism, which is a slightly different thing. Sallust saw the corruption of Roman government as the direct result of Rome's imperial success. Forgive the large block quotes (Sal. Cat. 10-11):

When Carthage, the rival of Rome's sway, had perished root and branch, and all seas and lands were open, then Fortune began to grow cruel and to bring confusion into all our affairs. Those who had found it easy to bear hardship and dangers, anxiety and adversity, found leisure and wealth, desirable under other circumstances, a burden and a curse. Hence the lust for money first, then for power, grew upon them; these were, I may say, the root of all evils. For avarice destroyed honour, integrity, and all other noble qualities; taught in their place insolence, cruelty, to neglect the gods, to set a price on everything. Ambition drove many men to become false; to have one thought locked in the breast, another ready on the tongue; to value friendships and enmities not on their merits but by the standard of self-interest, and to show a good front rather than a good heart. At first these vices grew slowly, from time to time they were punished; finally, when the disease had spread like a deadly plague, the state was changed and a government second to none in equity and excellence became cruel and intolerable....

Besides all this, Lucius Sulla, in order to secure the loyalty of the army which he led into Asia, had allowed it a luxury and license foreign to the manners of our forefathers; and in the intervals of leisure those charming and voluptuous lands had easily demoralized the warlike spirit of his soldiers. There it was that an army of the Roman people first learned to indulge in women and drink; to admire statues, paintings, and chased vases, to steal them from private houses and public places, to pillage shrines, and to desecrate everything, both sacred and profane.

He raises this point again in different terms in the Jugurtha (41):

For before the destruction of Carthage the people and senate of Rome together governed the republic peacefully and with moderation. There was no strife among the citizens either for glory or for power; fear of the enemy preserved the good morals of the state. But when the minds of the people were relieved of that dread, wantonness and arrogance naturally arose, vices which are fostered by prosperity. Thus the peace for which they had longed in time of adversity, after they had gained it proved to be more cruel and bitter than adversity itself.

This is the famous metus hostilis passage - the fear of an enemy maintained ancient Roman virtue. When that fear was removed, the people devolved into licence and debauchery. Sallust thought Rome's two great mistakes were destroying Carthage, the only power capable of challenging Rome's domination, and invading the East and bringing all the wealth back to Rome. He's not pacifist exactly; he would rather have Rome existing in a competitive Mediterranean international system instead of dominating it.

Dio, 56.33, has Augustus refusing to expand beyond the Rhine:

He advised them to be satisfied with their present possessions and under no conditions to wish to increase the empire to any greater dimensions. It would be hard to guard, he said, and this would lead to danger of their losing what was already theirs.

Clearly this is less about pacifism than it is administration. Also, Dio is late and the only one to report this, and we know Germanicus tried hard to get to the Elbe. Claudius got Mauritania and Britain, so on and so forth. If Augustus said this no one listened.

Some Romans were aware of barbarian (for lack of a better word) critiques of Roman imperialism. The most famous is perhaps from Tacitus' Agricola 30. This is a speech Tacitus puts in the mouth of the British commander Galgacus, which he gives to his troops just before fighting the Romans. This is how Tacitus imagines a barbarian talking to other barbarians about Roman imperialism, and not necessarily an opinion Tacitus expects his Roman readers to sympathize with, but it is such a great critique of Roman imperialism I can't believe Tacitus was completely impervious to it:

Now, however, the furthest limits of Britain are thrown open, and the unknown always passes for the marvellous. But there are no tribes beyond us, nothing indeed but waves and rocks, and the yet more terrible Romans, from whose oppression escape is vainly sought by obedience and submission. Robbers of the world, having by their universal plunder exhausted the land, they rifle the deep. If the enemy be rich, they are rapacious; if he be poor, they lust for dominion; neither the east nor the west has been able to satisfy them. Alone among men they covet with equal eagerness poverty and riches. To robbery, slaughter, plunder, they give the lying name of empire; they make a desert and call it peace.

So, I don't know of any out-and-out pacifists as in Gandhi, but Romans could be critical of their own imperialism and could imagine how others critique them.