I have read about the Salem Witch trials, but what I am wondering about it comparatively, how did people react to allegations of magic, witchcraft, and other local non-European customs of power (dances, sacrifices, etc.) in religious circumstances? What kind of interlacing with the secular powers happened when magic and witchcraft came into play in Spanish Colonial areas.
I ask in part because Catholicism in Mexico seems very much influenced by such rituals and I am wondering if they are a newer development or a long running effect from the melding of Spanish Catholicism with Aztec (and in Peru, Incan) mysticism.
For starters, there wasn't the same fear of "witchcraft" per se in Spain and Spanish America as in the British world.
Instead, the Spanish religious and civil authorities were primarily fearful of heresy and concerned with maintaining purity of religion.
The famous Spanish Inquisition was formed to police the observance of those claiming to be Catholic Christians, especially those who were converts or the descendants of recent converts. In both Spain and Portugal there was a fear that converted Jews or Muslims were still secretly practicing their old faith.
The fear wasn't so much that your secret Jewish neighbor was casting spells on you as it was rooted in the belief that to tolerate heretics among you was to invite the chastisement of God in the form of poor harvests, natural disasters, defeats in war, etc. This was rooted in medieval ideas of the body politic, in which the whole society is a single organism and must be united in all things in order to remain healthy and function properly. Likewise, there was no distinction between religion and politics, so heresy was a sort of treason and treason a sort of heresy.
In the Latin American context, the indigenous peoples were subjected to the same scrutiny based on fears (sometimes well-founded) that they were still practicing their pre-contact religions alongside their new Christianity.
So if you confessed that you were practicing "magic", it would be interpreted in this way, that you were still worshipping one or more pre-contact deities (which the Spaniards believed were idols at best, demons at worst) and were performing ceremonies invoking potentially demonic powers.
As to the consequences, it would vary considerably based on circumstances as well as the temperament of the priest to whom you confessed.
Given that you are confessing it, it would be assumed that you recognize your practice as a sin against God and are asking for forgiveness.
The priest might simply absolve you and later just keep a close eye on you, watching for signs of relapse. Or (and this was not uncommon) the required penance might include some form of public repudiation of your former beliefs and the rituals that went with them. This could be as mild as simply making a statement before the assembled village/town and burning any paraphernalia, or it could become a big spectacle where you'd be publicly humiliated in some manner (such as placed in stocks or pillory) and make a public recantation.
In answer to the second part of your question, the folk Catholicism of Mexico and other places in Latin America undoubtedly is rooted in various pre-contact beliefs and practices. The task of wiping out the preceding culture was not possible, so the Spanish missionaries allowed some aspects of pre-Christian practices to survive, provided they were properly Christianized.
For instance, a village might have an annual harvest festival in honor of a mother goddess. The festival would be allowed to continue but would be recast as a festival in honor of the Blessed Virgin Mary or some other Saint. Medicine and healing rituals once attributed to some local god or power would be reinterpreted as being accredited to the power of Jesus.
It's worth pointing out that this is not unique to Latin American Christianity. Take a look at the origins of the Christmas tree and you'll see that the same sort of negotiation between old entrenched beliefs and new religion took place in Europe.