Nowadays when you hear the word "mercenary" you either think of some PMC, former special forces security guard, some dodgy character who's spent time abroad and not fought for his country or poor men and boys from the 3rd world hired to suppress protestors at some dictators whim (Gaddafi, Libya 2011).
However, if you look back long enough, you can see frequent mentions of "mercenary armies", not just shady groups, hired to embark on campaigns rather than dirty work that you wouldn't ask regular soldiers to do. Seriously, from Egypt, to the wars of the Diadochi to the Punic wars, there seems to be a period where citizens took a back seat to mercenary forces in war. It doesn't seem like Carthage even used their own citizens in the Punic wars because of the frequent mentions of Greeks, Gallic mercenaries, Numidians and Iberians in their forces. Egypt even more so, using Libyans, Greeks, Galatians and Nubians. Rome had their system of auxilaries who had a vested interest in the long term of fighting for land rather than short term payment. And then of course there were special guard units of foreign warriors in the heart of the Roman Empire, such as the Palatine guards and the later Varangian guards in the Byzantine empire.
And even after the fall of the Roman Empire, mercenaries seem incredibly common, especially in Northern Europe, with mentions of "war-bands" of hired mercenaries in Saxon and Viking armies. And this keeps cropping up throughout the medieval era, with German and Dutch mercenaries sounding pretty common in the 15th and 16th centuries. I can't put an exact date on when fighting for wealth rather than national service drops off, but it seems like the 17th century might have been the century where this begins to take place.
So, what changed throughout this period for generals and kings and emperors to suddenly focus on building a professional force of their own citizens rather than a force of mercenaries? When did mercenaries stop being so common in war? Or are mercenaries still surprisingly common today and I just haven't noticed?
I'm not a historian, but I think the answer lies in the growth of nationalism as an organizing force for states and society; the changing nature of war; and the changing norms that outline behavior in the international state system (primarily the concept of neutrality).
Mercenary use was widespread in Europe (and thus internationally through international trade and empire) right into the 1800s in various ways, but it was not the only form of sanctioned violence carried out by non-state actors. By this I mean that until the 1800s (arguably later; I'm not a historian), it was not seen as an imperative for all legal violence to be carried out by the state. Chartered companies like the Dutch East India Company, for example, would have been semi-independent companies chartered by a European state government to engage in trade, establish and run colonies, engage in war with other companies or foreign governments/entities, etc., and they had the legal permission to engage in violent activities within parameters (often loosely enforced) and also hired mercenaries as needed. Privateers (legal pirates backed by a government) would be another example of an arrangement when states would sanction non-state violence. What makes mercenaries interesting is that in the centuries leading up to the 19th, they largely operated within an international marketplace that had no regard for national loyalties (though these loyalties of course did arise from time to time). A French king or lord may not think twice to hire Swiss, German, Scottish, or Italian mercenaries, and in many cases (especially in the lands of the Holy Roman Empire), these mercenaries may have had no choice, as they owed their services to their own local authority who had made the deal to sell their military labor.
The thing is, this whole arrangement kind of relies on the system lacking modern conceptions of loyalty, nationalism, and state authority. With the evolution of nationalism as a politically uniting force and the redefinition of the relationship between the state and its people, this became a difficult system to maintain. The levee en mass (mass conscription) introduced in the French Revolution seemed to be a game changer but an actual historian could give a better explanation. Before, wars were more often fought primarily with these small groups of professional armies, but that began to change, and with the rise of nationalism and the necessity of ruling elites to maintain legitimacy, putting down the masses with paid foreign mercenaries probably wouldn't be a good long term strategy.
All this tied into changing norms in interstate relations. If Britain and France are at war, can Bavaria really claim to be "neutral" (itself an evolving concept) if Bavarian mercenaries are fighting for one side or the other. What if the British government officially wants to back one side or stay "neutral" in revolutions in Spain's American colonies but thousands of British go to fight for money? There were certainly plenty of people in Europe, especially Britain, willing to sell their military labor (half-commissioned officers, reserves, veterans, etc.), how do you stop them? to give you an idea, the Barbary Pirates, who attacked European and American trading vessels off of the North Africa and sparked one of the US's first international crises, would have had a high percentage of Scottish and other white, Christian Europeans willingly working on the ships. These men would have had their own reasons, but many would have been acting our of pure economic self interest.
From what I understand, it seems like the use of mercenaries was in many ways preferred by ruling elites and many wanted to use them, but the drawbacks for central control of foreign policy/relationships and domestic legitimacy came to outweigh the benefits. Eventually, mass conscription and the changing nature of technology in warfare had an effect I'm sure, but I can't speak to any details.
I hope this was helpful, but Mercenaries, Pirates, and Sovereigns by Janice Thomson is an excellent and quick read if you'd like a good book on the subject,
I would like to point out you're completely underselling the importance and scope of mercenaries in the Byzantine Empire. By the time of the Komnenian war, it was often cheaper and more efficient for the Byzantines to hire foreign mercenaries and their own domestic ones to field their armies instead of the old farmer-soldier of the Themata.
I would say that Gustavus Adophus of Sweden in the early 1600's had a huge influence on the change from mercenaries to professionally raised, paid, and equipped armies. His disciplined forces worked together in what's arguably considered the first case of "modern" combined arms tactics. With all of his forces using the same caliber and type of weapons, it was cheaper to supply and arm rather than worrying about the logistics of obtaining different types of weapons for mercenaries. Additionally, he equipped his infantry with light artillery. In doing so, the infantry and their organic infantry trained and fought together to achieve maximum effect for an artillery bombardment followed by an immediate infantry attack. Thanks to his work, all of his forces worked as one machine, obtained through drilling, and his government was able to support the soldiers with their standardized equipment. He consistently defeated larger European armies in order to improve Sweden's status to a more dominant power.
Edit: typo