I'll focus on the first two decades up to 1520, and mostly on Mamluk response, as they were the principal actors at those moments, but as we will see Ottomans figure in those actions too.

First to paint the necessary larger picture of the period 1490-1500. The main Muslim powers were Mamluks in Egypt and Syria and Ottomans in Asia Minor and Balkan Europe, but around that time we also see the meteoric rise of Shah Ismail and the Safavid Iran. Those factions were interconnected in their fierce rivalries mixed with rare occasional joint cooperation. Mamluks internal situation was complicated at the moment of Portuguese arrival. The period between 1496 and 1501 was turbulent time of quick changing of Sultans, and areas like Jeddah and Yemen managed to only nominally remain subjects to the Sultan. When situation finally settled with ascesion of Al-Ashraf Qansuh al-Ghawri in 1501, the new Sultan was met with depleted treasury and a weak state with an army increasingly outdated. To alleviate himself he levied increased taxes on christian traders further rocking his already precarious relationship with European coming to Alexandria and Syria to trade, most important of which were Venetians. Special situation at the time involved diplomatic exchange between Spain and Mamluk Egypt, as after conquest of Granada in 1492 and particularly after rebellion of the Alpujarras in 1499-1501 bringing forced conversion of the Moors left in Castille, Moorish refugees asked the Mamluk Sultan to retaliate on Christian population and their sites in the Holy Land. In 1501 Spanish sent an ambassador Peter Martyr who successfully managed to negotiate no such retaliation occurred. This case is interesting as it shows some of the options available for Mamluks, and how they chose not to act upon them.

To get back to the reactions to the Portuguese, we have to keep in mind that Mamluks, despite sitting on key strategic position, themselves were little concerned with navies and maritime affairs and were content with huge income they were deriving from taxing the private merchants transporting merchandise through their territory. Vasco Da Gama's first arrival probably went unnoticed by them, given the humble nature of that first expedition, and the little impact it had. And if the Portuguese wanted to simply trade the Mamluks would be most probably similarly disinterested to make any action. However Portuguese from the very start had an incredibly aggressive attitude specifically targeting Arabs and Arabic ship. Already the second expedition, lead by Pedro Alvares Cabral, had Royal instructions to indiscriminately attack, seize and loot ships belonging to „Moors of Mecca“ meaning subjects of the Mamluk Sultan. The situation further deroded when in Indian town of Calicut tensions and struggle between Portuguese and local (Moorish) merchants in loading spices, escalated to a mob killing of 40 or so Portuguese men on shore, which in Portuguese mind was a declaration of open war.

When the fourth expedition to India was sent in 1502 (the third one was a small commercial venture which left before news of the second expedition's conflict came back), it was a full war force, with a squadron that was specifically assigned to the mouth of Red Sea with a task to capture ships and stop trade there. And Portuguese continued making those mission regularly for quite some time. Despite never managing to stop trade, the Portuguese did manage to capture many ships, in that earliest phase usually burning them and killing anyone left on board. Italian traveler Ludovico Varthema being incognito in Yemen around 1503 or 1504 mentions the effect these Portuguese attacks had, consequences of which he actually barely managed to avoid lynching by a mob of survivors who escaped from captured ships (who were trying to break into the prison where he was held to kill the Christians there)

The Portuguese attacks continued, and in 1505 they established a fort on Island of Socotra as a base to control access to Red Sea. The island proved not to be good enough for the task, and was abandoned in a few years time, but at that time it looked as their grip was tightening.

The Mamluks were forced to act. They were pressured by their own merchants, by outside forces like Ottomans and Venetians (who were also interestingly proposing building of a canal to link Red Sea with the Mediterranean), and in the end they were aware of the losses to their own income that would happen if the trade stops (which seemed possible in that early phase). However, even when they made up their mind they had problem of the logistical nature. They had no fleet in the Red Sea, nor was it really easy to build, equip and man one as they lacked forests, wood and experienced men.

In this, they were assisted by Ottomans (and clandestinely by Venetians) who sent wood from Black sea and men, and after huge effort, a fleet was assembled near Seuz, and sent in 1507. After delays in securing Jeddah, the fleet proceeded to India, where it met with allies (it was a very weak mistrusting alliance) coming from the city of Diu, adn they jointly scored a minor victory against a Portuguese squadron at Chaul in 1508. The Portuguese Viceroy Almeida, whose beloved son died at Chaul, responded by attacking and crushingly defeating this fleet at Diu in 1509.

Following this defeat, Arabic forces were unable to further challenge Portuguese forces in the Indian ocean, and in fact had to struggle to resist Portuguese attacks on the Red Sea which followed. In 1511, Ottoman Sultan Selim I sent men, gunpowder and firearms to help rebuild the Mamluk Red Sea flee and reinforce Jeddah and Mecca, a help which was crucial as Portuguese attacked Jeddah in 1513 and in 1517, both time unsuccessfully. Year 1517 was important as this was also the year when Ottomans conquered the Mamluk Sultanate in its entirety and took over the task of fighting the Portuguese. Initially, they too were fighting to keep the Portuguese from the Red Sea, but after the 1520s the conflict widened to the areas of Ethiopia, Yemen, Persian Gulf, and occasionally the far end of the Indian Ocean. In the local theaters the Ottomans were mostly successful, keeping the Red Sea free of Portuguese control, and after 1550s Portuguese altogether stopped the already ineffective expeditions to harass the Red Sea shipping. On the far away fronts, like India or Aceh Ottomans were able to provide some limited support in weapons or specialists to local forces fighting the Portuguese, but these occurrences were very rare and ultimately never really made much changes.