The short answer is basically yes, but that depends on what your definition of a "frontier" is. In northern South America (Ecuador, Colombia, Peru) the frontier would be the Amazon, while in Southern South America (Argentina, Chile), it would be the Patagonia. In both cases there are similarities with the American frontier: sparsely populated except by indigenous peoples until later, the control wasn't effective but just claimed for a long time, seen as a "savage" and somewhat dangerous land.
Here's where the difference comes in between the Amazon and the Wild West. Colombia, Ecuador and Peru essentialy saw the Amazon as a worthless, disease filled land, that they nonetheless claimed and had wars over. Their main efforts towards the region was not populating it with mestizos (the "primary culture" of those countries, so to speak) but to evangelize the natives, establish control and exploit resources. In that regard, the colonization of the Amazon was much more similar to the scramble for Africa than the colonization of the Wild West.
The catholic "misioneros" sought to bring God and enlightment to the natives. Some were considered savage animals, others were somewhat friendly. The misioneros, often priest with some soldiers and settlers, took upon themselves to penetrate into the deep Amazon, establishing cities for other misioneros and people going to other cities through the rivers of the region.
The Amazonian land was often poor, the natives were often hostile, the settlements were isolated. Thus, not many people wanted to go to the Amazonia. It wasn't seen as a land of oportunity. Some commerce did happen, but for the most part the central government forgot about these lands. The Ecuadorian Amazon for example was nominally organized into one big province, "Oriente", but the ones with actual political power were the misioneros there. Even today most of Peru's Amazon organized into a single, enormous department, Loreto (bigger than Germany).
It wasn't until abundant rubber and later oil reserves were found that people from more populated areas started to migrate to these old settlements, still populated mostly by catholic natives and priests. Thousands of skilled and unskilled workers arrived to work on these new businesses. Still, even today, the Amazonian provinces are lightly populated when compared with the Andean and coastal areas.
Now, the most similar thing to a frontier in South America is definitively the Patagonia. A large expanse of land just South of Chile and Argentina. Ripe for taking, the land was reasonably fertile and the governments claimed them and sent settlers there to establish effective control. Like the US, there were many natives that attacked the settlers, the climate wasn't as kind and the areas were only colonized from the 1850's onwards.
The Argentinian Law 1532 created in 1884 several "national territories" out of the mapuche lands of the Patagonia and a process of colonization started. Italians, Boers, Welsh people flocked to the territory and established farms, while protected by the army and accompanied by the misioneros. This was the "conquest of the desert". Like in the Amazon many conceived this conquest as a civilizing mission.
Chile started its colonization by the construction of Fort Bulnes in the straits of Magellan. The Selective Immigration Law of 1845 encouraged Germans to move into Southern Chile and colonize the land.
Sadly, these Argentinian and Chilean efforts to colonize their "Wild South" led to the extintion of certain native peoples.
Though perhaps not precisely a "frontier" like the American one, South America did have a certain notion of border regions, which they saw as savage places where they had to bring God and civilization.
Sources: El Proceso de Colonización de la Amazonia
u/Red_Galiray has already written about modern developments in Amazonia, which I'm less familiar with. So I'll focus mostly on colonial times, following the question of what a "frontier" might have meant in Spanish America - hopefully this can form a basis for comparisons with later periods and the U.S. frontier. I'll mostly draw on Tamar Herzog's recent "Frontiers of Possession: Spain and Portugal in Europe and the Americas".
Borders in Spanish America
As the title suggests, Herzog looks at space and borders in Spanish America via the conflicting claims of Spain and Portugal. She challenges more traditional notions of boundaries being set either by official actions (e.g. by rulers) or by treaties. Instead, local actions of people living in border areas and of communities are highlighted, meaning that borders were continually negotiated over centuries. This meant that chaos played a large role in how boundaries between Spanish and Portuguese domains were formed:
[...] contemporaries did not ask whether borders were linear or zonal, internal or external, artificial or natural. Instead, they mostly cared about the extension and nature of their usage rights. [...] In the process, rather than being predetermined and fixed, the meaning and extension of both “internal” and “external” [borders] were constantly reelaborated.
Starting with the Treaty of Tordesillas the lands not belonging to Christians were divided, with lands east of a meridian near the Cabo Verde Islands belonging to the Portuguese and those to the west to the Spanish. This marked the beginning of a centuries-long (and possibly ongoing) territorial conflict including judicial debates. Documents such as these papal bulls and other treaties failed to clearly define Iberian borders, in connection with a lack of certain geographic knowledge.
Border conflicts between Spain and Portugal included far-flung regions like the Moluccas and territories between modern-day Argentina and Uruguay. Even the union of crowns under Philipp II only strengthened the conflicts over time. After this
[...] the heartland of South America became a battleground. Portuguese from the east, Spaniards from the west, and ecclesiastics from both sides penetrated the interior with the aim of taking hold of both territory and its resources, people included. Because over time European penetration became more intense, by the late eighteenth century the dispute expanded dramatically to include territories that are presently part of Brazil, Paraguay, Uruguay, Argentina, Venezuela, Colombia, Ecuador, Peru, and Bolivia.
Various treaties tried to settle these issues in vain during the 18th century, so that they remained a source of further conflict between various powers even beyond the colonial period. Instead of treaties or settlements, it was the actual use or possession of territories by Spanish and Portuguese soldiers and missionaries that would form the eventual borders.
It's important to keep in mind here that different indigenous groups were often involved during these territorial conflicts. These followed their own interests, and bargained for their own good. Then again, Iberians generally described the lack of sedentary native activity in order to legitimize their own property rights. With the increasing focus on possession as a legitimizing narrative (rather than religious conversion forming the basis of conquest campaigns in early colonial times), traditional lands were increasingly appropriated or drawn into Spanish-Portuguese conflicts.
[...] both violence and peace were strategies used by the Spanish and the Portuguese in order to subject the indigenous peoples and [...] this subjection in turn had territorial implications. [...] [R]ather than a distinction between law and its application, what transpired in the American interior was an ideologically motivated divide between an internal and an external frontier, allowing actors to apply different criteria when dealing with rival Europeans and when facing natives.
To conclude
We're here of course still in earlier colonial times. But I'd still like to highlight the chaotic nature of the division between Spanish and Portuguese domains, which would continue to influence struggles over borders in the independent South American nations. After the Wars of Independence of the early 19th century, newly independent nations were formed, but many border disputes remained far from settled. So we have disputes over areas between Chile and Argentina (among many others) that reach into the early 20th century, that were only settled with the inclusion of European mediators and at a British court.
Trying to bring this back to your initial question: I would say following Herzog that in colonial times we can see the roots of an "unclear" or even chaotic way of installing borders, carrying over into modern times. I don't know enough about the U.S. "frontier" for a clear comparison, but still see some important differences here: a) Border disputes were held between European empires, and later South American nations - as opposed to a supposedly "open frontier" being pushed further west. And b) Conflicts with native populations formed an important part of the disputes, but not the main component - and native people could (at least in colonial times) still participate in or even profit in some ways from these territorial struggles.