In the 1500s the Spanish Tercios were some of the best organized, equipped, and disciplined soldiers in all of Europe. Truly a modern and capable fighting unit.
However they seemed to face a lot of struggles in the Thirty Years War, especially against Sweden. They seemed to lose relevance after that, until by the early 1700s they weren’t that formidable anymore.
What caused this gradual decline of the Tercio? Why couldn’t they modernize and keep with the times?
Your premise is basically flawed. The Tercio *was* modernized and kept up with time. That is precisely why it "disappeared", eventually being abolished even in name by the Spanish Bourbons in 1704, but make no mistake the tercios thus removed were nothing like the ones at the start.
It's important to realise that in many cases "tercio" is just equal to "regiment" in meaning. And e.g. the Spanish crown maintained Spanish tercios, but also German regiments and the difference between them was mostly one of nomenclature. Though the Spanish of course would regard Spanish troops (not entirely without reason) as better quality. This is not an unique position, the Swedish themselves (since you brought them up) during the same period had "regional regiments", fairly large units that served an administrative function in peace organising and training the levy in various parts of the realm.
There is also the further complication that the idea of "an outdated tercio against a modern Swedish pike&shot strategy" is basically 17th century propaganda particularly popular in Anglo-Saxon histography as a lot of English speaking sources were purposefully or unwittingly furthering the "Protestants are new and modern and Catholics backwards" ideas. The English at the time mainly fought with Protestants and particularly for the Swedes and thus took up this viewpoint in much of their writings. Basically a lot of the general viewpoints of tercios and the 30 year war is based more on propaganda rather than actual facts.
So what is a tercio? It's an administrative unit, ostensibly of three parts (hence the name it's theorized) of pikes, swordsmen and ranged troops. It traces it roots to the turn of the 16th century when the Spanish sought to create forces to imitate the so successful Swiss mercenaries that tended to fight for the French. The Holy Roman Empire took similar steps with the creation of the Landsknechts. The Italian Wars saw the Spanish develop both confidence and experience and tactics and strategy.
The tercios was never everything the Spanish crown could field, nor even a majority of it's troops. Though they tended to be the best. At their proper founding in 1536 a tercio in theory consisted of 3000 men, 10 pike companies and 2 of arquebusiers of 250 men, and I can't stress the "in theory" enough. By the 30YW period in question tercios already fielded more companies of fewer men, and some, despite being labelled as pike companies, contained fair numbers of ranged troops as well as pikes. The tercios followed the general trend in Europe of an increased ranged component over melee only troops. In 1632 it was standardized to 15 companies of 90 arquebusiers, 40 musketeers and 60 pikemen. And these companies were administrative, not tactical units. It required some effort from the officers before a battle to organise their men into tactical units, which is one reason we have battle plans that survive form the period. And manuals that helped you calculate how many men of various types you needed for certain frontages. At their peak of the late 1500s the Spanish tercios was most likely among the best troops that existed, possibly in history. Well motivated, very disciplined, well paid (well often at any rate, less so as we go on) and well trained and with cores of very veteran soldiers.
Did the tercios decline? Probably less so than that other "modern" armies were able to close the quality gap. At the height of their renown no other nation in Europe could afford to put so many trained men under arms and keep them there. And therein lies part of their perceived decline I think. The Spanish crown had so many commitments globally, so many wars, that the tercios were spread so thin they no longer enjoyed a substantial edge over other countries forces. Other forces may have started thinning out their formations' depths before the Spanish did, but it was because e.g. the Dutch rebels could not afford to keep as many troops as the Spanish (though in fairness eventually it turned out neither could the Spanish). Reducing the depth of files where men essentially did nothing in the last few ranks to expand frontages especially with more firepower was a winning concept and lead to the line of battle system in the 18th century. And by no means did the Spanish not follow this trend.
The Spanish did not per se "struggle against" other forces, they gave as good as they got. The Spanish/Imperial armies moved towards exactly the same type of forces as everyone else. Much have been made of the "Dutch system" or "Swedish system" vs "the old fashioned Tercios", but a lot of it is based on misunderstanding and misconceptions. Some of it is even based on a scrap of a surviving battle plan, a sketch of unit dispositions, supposedly used by the Spanish before a battle they lost, except that it wasn't. Arguably much of the 30 YW was fought using a "German system" developed over the war years what armchair military historians and wargamers lump as "pike and shot tactics".
Even the use of the solid infantry formations the Spanish supposedly used wasn't as bad as might have been surmised. Several battles ended with the Spanish infantry retreating in good order when their allies had already fled. A well motivated pike unit could fight to the last man but be ground down by artillery. The English Civil War even has an example of a pike unit holding out until effectively shot down to a man. With the growing prominence of field-artillery, that unlike at the beginnings of the tercios where not essentially bound to their initial positions (i.e. it became more feasible to move an redeploy cannons a they became lighter but no less powerful), massed infantry attacks tended to be moved down by firepower before they got going instead of pushing off the enemy from the field.
Basically over the 1500s and 1600s pikemen became less important as the core of military units until at the break between the 17/18th century pikemen became essentially obsolete. If we only define tercios in this general misconception as large inflexible pikeunits with some shot attached then they declined because military strategy moved towards more firepower in wider and not deeper formations. In reality though tercios developed over time as military strategy did and declined, if they did, more because the Spanish empire was exhausting military manpower and economic strength in endless wars and were finally abolished more as a break with a Habsburg royal past by the new Bourbon monarchy than anything else.
The Spanish Tercios 1536-1704 by Ignacio & Ivan Notario Lopez
Imperial Armies of the Thirty Year's war 1618-48 1 and 2 by Vladimir Brnardic
Pike and Shot Tactics 1590 - 1660 by Keith Roberts
and
The Army of Gustavus Adolphus 1 & 2 by Richard Brzezinksi
I particularly like this last one since it make short work with a lot of the English peddled myths about the Swedish army of the period and it supposed overthrow of tercios.
All from Osprey publishing, so relatively speaking fairly accessible and not too expensive reads on the subject.