There are 3 main parts of this question

The first is: "When did europeans find out that there were muslims in Indonesia?"

The second is: "When did they find out the extent of muslim presence in the area?"

And finally: "What was their reaction over it?"

As for the first, the answer is murky. Indonesia was where much of the spices that came into Europe ultimately came from, and for several centuries that trade was for the most part controlled by the muslim world. Muslim traders were present in indonesia since at least the 8th century, and it was common knowledge among european traders (particularly venice and genoa) that muslim traders traveled from "somewhere to the east" and brought back spices. So even if they had no first hand accounts, it's not unreasonable to say that europeans were aware that there was "some" muslim presence in india, indonesia however was a virtual unknown, when Pero da covilhã visited Calecut in the 1480's he was surprised to discover that a lot of the spices heading west, actually came from some place further east, carried by muslim merchants.

The extent of muslim presence in the region however was unknown until after the Portuguese reached india in 1498. Up until that time, the portuguese expeditions to india and ethiopia, looking for the mythical Prestor John had been unsucessful in transmiting information back to Lisbon. Some expeditions didn't manage to get past muslim controlled areas like egypt, and others, though they suceeded in going to india and ethiopia, like the voyages of afonso Paiva and Pero da Covilhã did not suceed in sending information back to Portugal.

It was only in 1511 with the conquest of Malacca that the portuguese would have better knowledge of the extent of muslim presence in indonesia. Up until then, it was believed (for the 10 or so years where the existence of indonesia was known to the portuguese) that the muslim presence was similar to what they experienced india, that is, a majority non-muslim population with some muslim traders or rulers.

As for the reaction of the Portuguese, there really wasn't much. Malacca was taken, and a few other forts were built in the spice islands, from where they could control the trade, and for the most part the strategy of getting a few forts from where to conduct trade, installing local kings friendly to them, and rellying on their naval power, thereby "replacing" the muslims role in the spice trade, was continued.

TLDR They basically said "huh" and carried on with their plans