While I can't speak to the private thoughts of the Spanish and Portuguese government officials, I can speak to how the border movement came to be.

Framework: Negotiation and Contained Conflict

The relations between Spain and Portugal in particular were often less "all-or-nothing competition" and more "contained competition and negotiation". Even before the Iberian Union, Spain and Portugal had already been entwined politically, culturally, religiously, and even dynastically for centuries. This was especially true in the colonies, as navigators, colonists, and sailors sometimes switched service between Crowns.

The first major post-Tordesillas competition was not in America but in Southeast Asia, as Tordesillas did not clarify an antimeridian for the demarcation line. In 1511 Ferdinand Magellan sailed to the Moluccas and claimed the islands for Portugal. Ten years later, Magellan sailed to the Philippines and claimed those islands for Spain during his famous circumnavigation voyage. Charles V of Spain contested Portuguese control of the Moluccas by sending expeditions of ships and soldiers after initial cartographic negotiations failed. Both Crowns wooed native rulers, and fighting broke out on the island of Tidore. The issue was resolved in the Treaty of Zaragoza (1529), where Portugal paid a large sum of money to Spain in exchange for Spain conceding control of the Moluccas.

This illustrates several points:

• While there was some violent conflict, the issue was resolved relatively quickly in a way that benefited both parties.
• Tordesillas was open to (re)interpretation.
• Flexibility in the colonies. Not only did Magellan claim lands for both Crowns, but he did so despite being a Portuguese subject. Meanwhile after Spain conceded the Moluccas, the Spanish governor of Tidore vacated Asia by taking a ship to Lisbon.

The Iberian Union

Due to dynastic intertwining, the King of Spain also became King of Portugal after the extinction of the House of Aviz. This is known as the Iberian Union when the whole of Iberia was united under one monarch. While Portugal and the Portuguese colonial possessions were still governed separately from the Spanish Empire, this put much of intercolonial conflict into abeyance.

Now merchants and expeditions could move even more freely between the two realms, as they were now joined under one monarch. Even in Iberia, Portuguese slavers routinely offered Amerindian slaves to Spaniards willing to cross the porous border (at this time Amerindian slavery was illegal in the Spanish Empire, while it was legal in the Portuguese Empire). Portuguese slavers sometimes even crossed into Spain itself to sell Brazilian slaves.

Meanwhile in South America, the Portuguese expeditionary Pedro de Texeira travelled under the auspices of Portugal up the Amazon river, eventually arriving in Spanish Quito. He charted much of Amazonia, and opened the region up to both settlers and slavers who would come to penetrate deep into the interior of South America. This would inaugurate a stronger pattern for the next century: while the Spanish remained in Andean South America, the Portuguese began to fan west into the Amazonian hinterlands to both establish settlements and capture Amerindian slaves.

Imperial Competition

Just as important, the Iberian Union opened up Portugal to greater competition from other European powers, as Portugal now became "fair game" for the rivals of Spain. Both Spain and Portugal were subject to incursions from other European powers.

In the 1600s the Dutch began competing with Portugal in Southeast Asia, fighting for control of the islands of Solor and Timor - and eventually establishing the Dutch East Indies. This competition expanded into South America, with the Dutch capturing and quickly losing Bahia in 1624. The Dutch launched a renewed invasion in 1630 and this time they conquered much of eastern Brazil, ruling the region until 1654. It was during this period that the Dutch also established their colonies in the Guyana Basin, which would prove much longer-lived (and which today are the countries of Suriname and Guyana).

The French presented a different colonial challenge to the Iberian powers, as despite being Catholic they never recognized the papal grant made to the Iberian crowns. After two failed attempts at establishing settlements in Brazil (one at Sao Luis and one at Maranhao), in the 1600s they established a colony in what is still today French Guiana, directly next to Portuguese Brazil. At the start of the 1700s they founded New Orleans and Louisiana, dividing Spanish Florida from Spanish Texas.

Meanwhile, in the 1600s the British (and the French and Dutch) began conquering Caribbean islands that were nominally Spanish possessions. The British also began pressing south into Georgia, abutting into what was formerly in the Spanish sphere of influence.

Treaty of Madrid

By the 18th century, both Spain and Portugal were seeking to reform their governments and how they related to their colonial possessions. In Spain these were called the Bourbon Reforms, where under the new Bourbon dynasty the Spanish Crown sought to rationalize the somewhat helter-skelter American dominions and promote their overall stability.

In Portugal reform was pushed by the Marquis de Pombal, who was chief minister from 1750 - 1777. In 1750 he negotiated the Treaty of Madrid, in which Spain conceded much of central South America (i.e. Amazonia) to Portugal in exchange for Portugal conceding territory around what is today Uruguay and Argentina. This abrogated the line of Tordesillas in favor of the principle of uti possidetis - meaning that each Crown would gain title to what it already held.

The specifics were renegotiated through several other treaties, but at least in regards to Brazil this more-or-less held firm. From the Spanish perspective, they conceded territory that they never effectively controlled to begin with while gaining title to territory near their Río Plata colonies. (In one of the later treaties, they even received what is now Guinea Equatorial, becoming their first outpost in sub-Saharan Africa). At least legally, this settled the affairs of the South American interior, which provided assurance that other Europeans such as the French and Dutch would be unable to make further incursions and allowing both powers to focus on security concerns elsewhere.

Sources

Durand, Frédéric. History of Timore-Leste

van Deusen, Nancy. Global Indios: The Indigenous Struggle for Justice in 16th Century Spain

Fausto, Boris and Fausto, Sergio. A Concise History of Brazil

Meade, Teresa. Brief History of Brazil

Plotkin, Mark. The Amazon: What Everyone Needs to Know