Considering its success in the Philippines where Islam had already been present and all of South America, how come Indonesia remained mostly Muslim?
I can't answer regarding South America, but I can give a little insight into the situation in maritime Southeast Asia.
Christianity, both Catholicism and Protestantism, arrived in maritime Southeast Asia with the coming of the Europeans. 4 colonial powers established themselves in the area - the Portuguese, the Spanish, the Dutch and the British. Whether a colonial power succeeded in converting large numbers of locals to Christianity was roughly dependent on three factors:
Thus, the spread of Christianity was dependent on far more than just missionary work. A brief look at the three factors:
Evangelism intent
In the second half of the 15th century, the papacy issued a series of papal bulls that would guide Catholic nations Spain and Portugal in their colonial activities. The two main points relevant to this answer were:
The Bull Inter Caetera of May 4 1493, for instance, says
Moreover we command you in virtue of holy obedience that, employing all due diligence in the premises, as you promise, nor do we doubt your compliance therein to the best of your loyalty and royal greatness of spirit, you send to the aforesaid mainlands and islands worthy, God fearing, learned, skilled, and experienced men, in order to instruct the aforesaid inhabitants and dwellers therein in the Catholic faith and train them in good morals.
Attaching this condition meant that if Spain and/or Portugal were found guilty of failing to evangelise, their rights to colonise granted by the papacy could be revoked. This threat was taken extremely seriously by King Philip II, who was, according to the first Bishop of Manila,
… so Catholic and Christian a prince that [he] would undertake nothing unless first convinced of its just and legal execution.
Thus, when Miguel Legazpi set out to colonise the Philippines in 1564, his expedition included 5 Augustinian friars. His instructions stressed that
His Majesty’s most important intention is the advance of our Holy Catholic Faith and the salvation of the souls of those heathens.
We can also see how keen the king was to avoid wholesale slaughter and enslavement in his other injunctions, which stressed that the Spaniards should settle far from anywhere occupied by the natives, secure their friendship through peaceful means and teach them a civilised way of life.
So desperate was King Philip II to colonise the Philippines justly and legally, that on reaching the Philippines, Legaspi expressly sought the opinion of the Augustinian friars as to whether it was right to proceed with settling the islands. And, in 1572, the Augustinians were once again asked for their formal religious opinion on whether the colonisation of the Philippines was “just” (interestingly, the Augustinians unequivocally responded that Spain had “no just title” to the islands).
Evangelism was thus baked into the Spanish mode of colonisation, giving rise to the popular description of Spanish colonisation as being for "God, Gold and Glory". Indeed, the Philippines was never able to generate revenue for Spain, surviving instead on an annual subsidy, and as the hopes of riches faded, conversion became the primary justification for the colonisation of the Philippines.
Dutch colonisation, on the other hand, was initiated not by the state but by the Dutch East India Company (VOC). Being a company, its aims could be summarised as being "profit, profit, profit", and then some more profit. While there were some in Amsterdam who viewed Southeast Asia as an extension of the battleground between Protestantism and Catholicism, the VOC's intent was simply to control the spice trade and make obscene amounts of money. When they did engage in missionary activities, it was generally with some kind of profit motive in mind, rather than it being an end in itself.
Location
Christianity came relatively late to the region. Hinduism and Buddhism had arrived in the 4th century, Islam in the 9th century. These had not only spread among the local population, but had been adopted by local rulers and were intertwined with political systems in the region. When the Portuguese arrived in 1509, maritime Southeast Asia was dominated by Muslim polities, with some Hindu polities remaining in parts of what is now Indonesia. This led to three obstacles for any missionary work.
Among the population, attempts to convert were met with resistance from existing religious structures. In the Islamic sultanates, for example, imams helped to maintain the religion through preaching. There were also community events linked to Islam, such as the celebration of the circumcision of a local leader's son. So it wasn’t like someone could just rock up to a missionary, get baptised and go home as if nothing else had changed.
In addition, rulers had used religion to legitimise their positions, much in the way Christianity played a major role in the legitimisation of a European monarch. They were not likely to encourage the spread of Christianity, or even give permission for missionary work to commence.
Finally, these polities were no pushovers. They were powerful in their own rights, had sophisticated organisation and diplomatic relations. A foreign power trying to spread religion through or after conquest soon found itself facing far reaching consequences. Portugal, for example, conquered the wealthy Sultanate of Malacca in 1511. However, the sultanate had had excellent relations with Ming China, which retaliated by rejecting a Portuguese embassy, crushing Portuguese attempts to establish formal relations with China. In the region itself, the survivors of the royal family regrouped and launched attacks on the Portuguese, easily finding support from other Muslim rulers. While the Portuguese managed to hang on to Malacca, they were never able to expand beyond the city itself, and the colony spent the next 130 years or so surrounded by angry, hostile Muslim states. Under those circumstances, it was unsurprising that evangelism in Malacca was not very high on the list of priorities (although there were certainly other reasons as well).
Where the spread of Christianity was most successful was in areas where other organised religions had not yet taken a strong hold, and where the political landscape was much more fragmented. Unsurprisingly, this was at the very edges of maritime Southeast Asia. In the southeast, Portuguese and Dutch missionaries had success in Timor - the roughly analogous Indonesian province of Nusa Tenggara Timor is today 60% Catholic and 35% Protestant - and Papua. And in the northeast, the Spanish had much success in Luzon and the Visasyas, while never quite being able to "pacify" the Muslim polities in Mindanao.
(continued in reply)