I use the Portuguese as my example as they were the pioneers of the age of exploration. Also the first explorers with a more structured system.

Was there a written protocol, no. But, the intent of the, typically monarch-backed, explorers were to create trade routes... for trade and to enrich, empower their backer. Religion was not their priority and followed in their footsteps. So, the intent of most explorers was to create relationships with the new groups that they met, thereby allowing them to trade and receive assistance in resupplying and repairing their ships. If the port or people were hostile or blocked them. The Portuguese would typically just kill them and/or take over the port (Calicut) and use it as a logistics point to benefit future ships. This rarely occurred though. They did however buy and/or settle their own forts/ports as well. Typically a port would be friendly because the ruling/aristocrat/upper class understood that this would increase their wealth and influence. If there were ever violence towards the Portuguese explorers, they usually went overboard and would just decimate the people for it. There is one account of Arab traders getting the locals angry against the Portuguese in Calicut region and the locals killed several of the Portuguese factory workers. Cabral was there, became irate, and captured a dozen Arab trade ships, killed all of the crews, took the cargo, burned the ships, killed the locals... yes. If I remember correctly, after that, they didn't experience many issues and eventually relinquished Calicut from their control because the region had warmed up to them, relaxed, and was in their best long-term interest for the region.

edit: in the example of China, the Portuguese had much more tact and made it clear that trade was the priority. China had closed most of their trading with foreigners and did not like having outsiders enter their kingdom (xenophobia). The Portuguese solved some of the local problems, mainly removing the local pirates for their fishermen. This improved the situation to where eventually China started to trade with them.

I hope this gives you an idea on their unwritten protocol.