We have a couple of detailed accounts of settlements being sacked by Greek mercenary forces, which gives us an impression of what these events would have looked like on the ground.

The most reliable is Xenophon's eyewitness account of the capture of a fortified town of the Drilai in Cappadocia by a force of Greek and Thracian mercenaries cut adrift by the death of their paymaster in 401 BC. These mercenaries were making their way home, relying on plunder for supply, but also looking to pick up what riches they could. Traversing the territory of the Drilai, they soon identified a vulnerable place and planned their assault (Xenophon, Anabasis 5.2.13-27). The attackers first unleashed a barrage of arrows, javelins and stones on the battlements, causing the defenders to abandon their positions. That signalled the beginning of the assault proper:

Agasias the Stymphalian, putting aside his arms and wearing only his tunic, climbed up, then pulled up another man, and meanwhile another had made the climb, so that the capture of the stronghold was accomplished, as it seemed. Thereupon the peltasts and the light troops rushed in and proceeded to snatch whatever plunder each of them could get.

Xenophon himself, however, who commanded the mercenaries, noticed that the defenders were gathering in their citadel and preparing a counterattack, so he held back most of the hoplites at the gate as they tried to get into the town. It soon turned out that his caution was justified:

After no long interval a shout arose within and men came pouring forth in flight, some carrying with them what they had seized, then soon a number of men that were wounded; and there was a deal of pushing about the gates.

But he questioned the fleeing men and found that the force mounting the counterattack was not very impressive. He figured his men would be able to handle the challenge, and stopped holding them back:

Then Xenophon ordered Tolmides the herald to proclaim that whoever wanted to get any plunder should go in. At that many proceeded to rush into the gates, and the crowd that was pushing in overcame the crowd that was tumbling out and shut up the enemy again in their citadel. So everything outside the citadel was seized and carried off by the Greeks.

The town of the Drilai was sacked and burned to the ground. Most of the people got away unscathed, however, because the Greeks could find no way to assault their citadel. The situation was very different in the Phoenician settlement of Motya on Sicily just 4-5 years later, when a Greek coalition army under Dionysios of Syracuse completed a mole across the bay to their island city and stormed it with siege towers. There was no safe haven left in Motya, and so the inhabitants defended their homes to the death. For days they withstood the Greek onslaught, fighting in the alleys and on the roofs of their houses. When a night attack finally broke their resolve, the scene that unfolded was horrific:

Dionysios' entire army burst into the city, coming also by the mole, and now every spot was a scene of mass slaughter; for the Sicilian Greeks, eager to return cruelty for cruelty, slew everyone they encountered, sparing without distinction not a child, not a woman, not an elder. Dionysios, wishing to sell the inhabitants into slavery for the money he could gather, at first attempted to restrain the soldiers from murdering the captives, but when no one paid any attention to him and he saw that the fury of the Sicilian Greeks was not to be controlled, he stationed heralds to cry aloud and tell the Motyans to take refuge in the temples which were revered by the Greeks. When this was done, the soldiers ceased their slaughter and turned to looting the property; and the plunder yielded much silver and not a little gold, as well as costly raiment and an abundance on every other product of felicity. The city was given over by Dionysios to the soldiers to plunder, since he wished to whet their appetites for future encounters.

-- Diodoros 14.53.1-3

I'm citing these sources at length because they give a real sense of the chaos. Both among the Drilai and at Motya, the scene is one of soldiers essentially running amok, often ignoring their officers, disregarding their own safety to satisfy their urge to steal and destroy. The attacking army would break down into uncontrollable masses running this way and that. Buildings and walls would be torn down to allow safer entry and retreat; fire would be spread either by accident or on purpose; the people in the city, terrified and desperate, would either mount their last hopeless defence or do what they could to avoid becoming a victim. There was practically nothing a commander could really do to control any of this. He was generally reduced to unleashing his men and letting them do as they wished - especially if he wanted them to believe that service under him would be worth their while.

This is the context we need in order to answer your question. Within this grotesque spectacle of violence and destruction, who would notice an individual soldier who refused to rape and pillage? Who would even see that soldier, let alone care that he was not taking part? In order to be noticed, he would practically have to go around screaming "Look at me! I am not killing or stealing! I refuse to kill or steal!" And even then, most of his fellow mercenaries would probably be too focused on getting their own share of the loot to give our conscientious objector a second thought.

Indeed, in a typical sack there would probably be many soldiers who effectively didn't take part - either because they were too concerned about the enemy, because they were being held back on purpose, or because they simply didn't have the means to get into the city before everyone and everything inside had been captured or destroyed. As Xenophon says, many went in at the call of the herald - but not all. Some would keep themselves out or be left out. The only response they were likely to get from their fellow mercenaries was "better luck next time".

On the other hand, since many Greek mercenaries had to go without pay (or served on promised pay) for long periods, it would have been difficult for a mercenary to survive if he refused to steal. How else was he going to pay for his grain? There were no supply officers in Greek armies, and it was not considered a commander's duty to supply the troops - merely to supply them with a market where they could buy their own food and other necessaries. Mercenaries were therefore in constant need of either coin or human or material possessions they could sell in return for coin. So while many troops might be left out of the sack of a city or town on each occasion that a town was sacked, it would be extremely unlikely that any Greek mercenary would refuse to take part in such actions on principle. The less opportunity he had to take part in a sack and enrich himself, the more keen he would be for the next chance to pillage. If your commander was the type to try to "live off the land", that would require you, too, to support yourself with plunder. And there was nothing wrong with that in the eyes of the Greeks; after all,

It is a law established for all time among all men that when a city is taken in war, the persons and the property of the inhabitants belong to the captors.

-- Xenophon, Education of Kyros 7.5.73

In short, if you refused to plunder or rape on one occasion, your companions probably wouldn't notice in the midst of all the chaos that accompanied the sack of a city. But if you stuck to your refusal as a matter of conscience, you would likely starve, and the fellow mercenaries who buried your remains would probably think you made a very strange choice the day you decided to become a mercenary.