What was actually stopping Hamiclar or Hannibal Barca from taking their army and overthrowing the republic in Carthage?
I'm going to mostly focus on the Carthaginian army on this matter as it is the one that I have a greater percentage of the full understanding on compared to the late Roman Republic and Roman Empire.The Carthaginian Army is often painted with the Brush of being an almost entirely "mercenary" force. However this is a large generalization, mostly given to us by the Romans, who call these non-citizen troops mercenaries. The non-Carthaginian forces that constituted the Carthaginian army came from a wide variety of sources and backgrounds, these troops were not simply mercenaries and this is a gross simplification. The Carthaginians acquired these troops through various means, and the different ethnicity that made up the army came to the army through different circumstances.Firstly lets get the dictionary definition of Mercenary- " hired for service in the army of a foreign country " or " one that serves merely for wages " are the two definitions in Merriam Webster, now these definitions are a bit simplistic, but lets use them for the purposes of this exercise. Lets look at each of the 'most common' ethnic groups who show up in the Carthaginian army individually to see how they would arrive in Carthaginian service, and whether they fit this definition.
Libyans- the Libyan infantry and cavalry supplied to Carthage were supplied from cities and peoples who were subjects of Carthage. The Libyans had a similar position in Carthage as the other Italian cities had with Rome. They were more directly controlled by Carthage and in the punic cities were simply a lower class, and so the Libyans raised were effectively a levy. These men were not mercenaries but soldiers raised from the population of the state. This doesn't fit either of our definitions of a mercenary, the Libyan's are not fighting for a foreign country, and they are not fighting for wages but out of obligation
Numidians - The Numidian came from allies or client states (moreso client states in the Numidian case) of Carthage. Again these would be troops supplied to Carthage as part of the terms of a treaty or alliance between the particular Numidian clan, and the Carthaginians. This again doesn't fit with our definition or idea of mercenaries. The Numidians are being requisitioned for service as part of alliance or treaty obligations, so they aren't really mercenaries, furthermore while you could technically class Carthage as a foreign state, it is one in alliance or an overlord over the Numidians own state, which blurs the lines on that front as well. Some Numidians probably were mercenaries, but this was the exception not the norm.
The Spanish- The Spanish are a little more complicated. For instance at the Battle of the Great Plains Livy and Polybius both use the term mercenaries when describing the 4,000 Celtiberians in Hasdrubals service. This is also used in the description of the 20,000 Celtiberians in Roman service at the Upper Baetis. However at the Upper Baetis for instance the Iberians under Andobalas in Carthaginian service are mentioned as allies. When Hannibal is mustering his great army for the conquest of Saguntum on north Eastern Spain (and his later march on Italy) he calls together troops from all his subject tribes. Why would you 'muster' mercenaries? We must also be careful with our sources as the Romans are quick to describe any non-citizen troops as mercenaries. So some of these troops were mercenaries, however a larger proportion seem to be similar to the Numidians, troops supplied by allied or client states of carthage as part of the terms of treaties or at carthaginian request.
Balearic slingers- We know little about the relationship between the Balearic slingers and the carthaginians, however the Islands had been under punic control for a very long time, so again I suspect these soldiers were again men raised or payed from directly by their Carthaginian overlords.
Gauls & assorted others- The Gauls, Greeks and others who entered Carthaginian service are perhaps the troops who could be best described as mercenaries, these were people with no clear affiliation with carthage, they were not subjects or allies, and they fought for pay specifically, and were hired instead of levied, mustered, or conscripted.
So we can see that the Composition of the non-Carthaginian members of the Carthaginian army were not merely just mercenaries but came about through a variety of different relationships. This is important because it means that not only do these people have loyalty to more than just money, i.e the army, that loyalty is related to both Carthaginian and non Carthaginian factors.
Now we get into the more speculative area of this question. First why did the Carthaginian troops not rebel for their generals? Well firstly precisely the fact that they have no relationship with Carthage. What benefit do they have to having "their man" in charge? Their families are at home, apart from the Libyans most of them are not under direct/semi-direct Carthaginian control. Unless they plan to accept citizenship and move their family they weren't going to get anything out of it. Furthermore they had their own interests at home, many of these men were leaders and nobles at home, they were here out of obligation to Carthage, not because they have a vested interest in Carthage itself. They had their own politics to care about before they started with Carthaginian politics
By Contrast the Roman troops were professionals/semi-professionals, they almost always served in multiple campaigns with their leaders for several years. They were also citizens of Rome, they had a vested interest in the politics of the Republic and they had much to gain from supporting their leader for office.
Leading on from this, most of these troops would return back to their own nations on completion of their service, Carthaginian armies were 'in general' not permanent professional forces, but troops raised for specific campaigns and then dispersed with their pay after.
Thirdly from a leadership perspective most of the Carthaginian leaders (especally in the case of Hannibal and Hamilcar) were too busy with their conflicts with their enemies to have much time to think about overthrowing Carthage, they were fighting for Carthage why would they overthrow it?
By Contrast many of these Roman generals such as Caesar in the late Republic were on personal missions, often against senate wishes, to achieve greater personal wealth and power. Caesar wasn't out fighting Gauls for Rome but for himself.
However this doesn't mean that the Carthaginian troops were never loyal to their leader above the state. I mean Hamilcar effectively established his own semi-independent Kingdom in Iberia and we don't have any records of his troops objecting to it. During the mercenary wars the Carthaginian officer Gisgo negotiated with the mercenaries because of his prior relationship with them, partly because Hamilcar was worried they'd kill him if he went, but also because they were far more likely to come to terms with their leader than some Carthaginian senator.