In r/worldbuilding we're discussing a mercenary based Military for a merchant state and someone mentioned that it was an guaranteed coup but I can't think of any historical examples
That is a difficult question to answer, not because we don't have the necessary information to answer your question, but because there is an enormous definitional and semantic issue. It's also an example/trivia seeking question, but it hasn't yet been taken down so we can pretend the question's asking, "Were Renaissance mercenaries able to stage a coup d'état?" Which is slightly better, but it's still very vague and open (Which renaissance mercenaries would you be talking about?) giving me an open field to ramble incessantly.
So the first issue is a, possibly minor, issue with periodization. The second and more pressing issue is tied to the fact that for the most part, pre-modern European states did not have standing armies, making the term "mercenary" problematic.
In pre-modern Europe, the standard procedure for military recruitment involved the award of a commission on a captain who would oversee the recruitment and training of an armed force. Some governments emitted many individual commissions, others commissioned a single Captain-general who would be expected to oversee the majority of the war effort. There were seldom any requirements tied on the nationality of the recipients of these commissions, nor were there any requirements imposed on the nationality of the people the officers recruited. In fact, if a foreign war had recently concluded, very often that war's veterans would soon be in demand nearby; Renaissance rulers didn't care where readily trained and battle-tested soldiers came from. Although that's not entirely true: some statesmen of the time commented on the benefits of recruiting locally rather than casting the net wider; Machiavelli, famously, advocated in vain for what he envisioned as a Roman-style citizen militia.
So although the pre-modern system of military recruitment is largely aligned with our conception of what a mercenary is, europeans at the time would not make that distinction. This is especially evident in Italy; the archaic word Condottiero is often translated as "Mercenary Captain" (even by Italian scholars!) is more accurately rendered as "Contractor;" as in, a person who signs a contract with the government (from the root Condotta, meaning "Contract," predictably).
As we move farther back, the apparatus of government simplifies, and rulers are more reliant on the local landed aristocracy. But even in the supposed age of chivalry, soldiers opportunistically backed different sides based on plunder, risk/reward, and who seemed to be winning. Many men who had been born into landowning families stood to inherit little more than a horse and a name, and were quick to attach themselves to powerful landed lords, offering their service in battle in exchange for lodgings or at the very least a place in their retinue.
That being said, I can think of two situations when military commanders who were distinctly outsiders were able to topple systems of government. But the mechanism by which they did is more complicated than you might think. The first involves Norman adventurers in Southern Italy in the 11th century, the second has to do with an Emilian-born and Neapolitan-raised leader active Lombardy in the 15th. Both are characterized by very different events.
The Normans, from eleventh-century northern France, perhaps as a semi-conscious hark to their Viking forefathers were particularly predisposed to offering their services as bodyguards to people traveling to the holy land (which would be mostly clergymen and nobility at this point). When they invariably passed through the southern half of Italy, they found the Principality of Salerno (a rump-state of the defunct Kingdom of Lombardy) and about a half dozen splinter Duchies locked in systemic conflict with both the Byzantine Empire and the Abbasid Caliphate. Soon, a small clique of about four of these Normans, with retinues that could be as small as thirty followers (at least according to the Byzantine chronicler Anna Comnenos) entered the service of Guaimario, the Prince of Salerno. Before long, ethnic Normans were an important component of the Salernitan armed forces, contributing to the stabilization of Guaimario's relationship with his subjects. The Norman brothers William, Roger, Drogo, and Robert carved for themselves a semi-autonomous fief in the Italian region of Puglia, seized from the Byzantine Empire. When in 1047 Guaimario was assassinated on the orders of the a particularly rambunctious subject, it would be the Normans in Puglia to rush to Salerno in order to place Guaimario's son Gisulfo on the throne in place of the conspirators' candidate. But Gisulfo's relationship with his subjects in Puglia soon grew strained for not entirely clear reasons beyond the desire to seize more power for himself. All would come to a head in 1053, when Gisulfo struck an alliance with the emergent Papal States to the north in order to organize a punitive expedition in Puglia. The outcome was a resounding defeat, causing Gisulfo to retrench himself in Salerno. By 1076 the Normans had removed Gisulfo from power altogether, along the way having seized Calabria from the Byzantines.
The Normans would go on to conquer the whole of Sicily, paving the way for Roger's son (also called Roger) to be crowned King of Sicily by the Pope. A rather dramatic ascendance for these Normans. But what's the point, beyond the fact that u/AlviseFalier likes to ramble about obscure Italians who lived close to a thousand years ago? Well, what I'm trying to transmit is that the Normans first integrated themselves in the land-based political hierarchy in Southern Italy, and once they built a power base for themselves, they took advantage of the disunited and self-destructive political class. So in theory, they are an example of a pre-modern mercenary coup, but are they even mercenaries, or are they simply migrants? Would they have removed Gisulfo from power if he hadn't turned against them? Had Guaimario not been murdered, would the Normans even accrued as much power as they did under his short-sighted successor? The conditions for revolt are unique are less clean cut than you might think. If you'd like to read more about what happened in Southern Italy, John Julius Norwich (my absolute favorite italianist-by-way-of-byzantiuim) wrote a history in two parts, titled The Other Conquest and The Kingdom in the Sun respectively.
The second example is that of Francesco Sforza. A bit of background: In the 14th century, Italian states begin transitioning to a more systemic pseudo-professionalization of armed forces; which sometimes could balloon out of control. Armed bands of men with few ties to the land could quickly become problematic if they were milling about; and during lulls in fighting or truces bands of armed men could disperse all over the country. Far from home, they pay had no reason to exhibit restraint, especially when their pay was less than regular. Indeed, for the most part companies of professional soldiers were unemployed veterans who moved swiftly from place to place, and suffered severe desertions if they could devise no ransom or find no employer (although they also faced desertions when they were employed; during the Savoy-Monferrat conflict in the mid 14th century the Savoyards withdrew to the mountains to force a stalemate, and several hundred english soldiers got bored and packed up to enlist for garrison duty in Crete on behalf of the Venetians). This certainly contributed to creating a vicious cycle by which larger, richer cities were constantly hounding out weaker ones; between the 14th and 15th centuries, the Milanese, Paduans, Veronese and Florentines were almost continuously involved in some kind low-level warfare, often bullying smaller towns nearby into submitting to their growing states.