To put it simply, the reason Portugal did not invoke Article 5 of the NATO Treaty (the article about collective defense) is because it could not, per Article 6:

"For the purpose of Article 5, an armed attack on one or more of the Parties is deemed to include an armed attack:

• on the territory of any of the Parties in Europe or North America, on the Algerian Departments of France 2, on the territory of Turkey or on the Islands under the jurisdiction of any of the Parties in the North Atlantic area north of the Tropic of Cancer;
• on the forces, vessels, or aircraft of any of the Parties, when in or over these territories or any other area in Europe in which occupation forces of any of the Parties were stationed on the date when the Treaty entered into force or the Mediterranean Sea or the North Atlantic area north of the Tropic of Cancer.

The only colonial possessions considered covered by NATO were the French possessions in Algeria, as they were legally considered part of Metropolitan France.

As for international reactions: broadly speaking, they did break down along Cold War lines. Soviet First Secretary Khrushchev was actually in India on a state visit when hostilities broke out on December 18, and he made it publicly clear that he considered Indian actions to be "absolutely lawful and justified", and that the Soviet people stood behind Indian actions. The Soviet embassador at the UN would later go on to state that has far as the USSR was concerned, it was an internal Indian matter. This position was echoed by other Soviet bloc countries in Eastern Europe, plus Yugoslavia.

Britain and Commonwealth countries like New Zealand, Australia, and Canada struck a rather cautious tone. Britain itself felt stuck between supporting long-standing ally Portugal and alienating a major Commonwealth member such as India. It and the other countries described generally took a stance of shock/surprise/regret that the situation was being resolved by military means, while also recognizing the strong Indian public support for transferring the Portuguese territories to Indian control.

The United States similarly took a legalistic stance of concern over the use of force. At the United Nations, US ambassador Adlai Stevenson would be a bit stronger in language. He first praised India's role in the United Nations ("Few nations have done more to uphold the principles of the Organization or to support its peace-making efforts all over the world"), and noted that he considered Prime Minister Nehru to be a personal friend, and also recognized the strength of Indian feeling on the Goa issue. Yet he went on to decry the use of force to resolve a territorial dispute, and argued that it jeopardized the UN Charter. He sponsored a Security Council resolution calling for a cease-fire, Indian withdrawal and a negotiated settlement, but the Council voted 7 in favor, 4 against, with the USSR casting its 99th veto to sink the resolution.

A number of former colonial possessions were much more vocal in support of India, including Indonesia, Ceylon, Egypt and Ghana. India's Nehru, Indonesia's Sukarno, Ghana's Nkrumah, and Egypt's Nasser had, along with Yugoslavia's Tito, established the Non-Aligned Movement earlier that year, so it makes sense that these states in particular would come out strongly in favor of India. In contrast, Pakistan (perhaps unsurprisingly) denounced the "naked militarism" of India, while maintaining a neutrality in the conflict to the point of interning Portuguese ships and confiscating Portuguese ammunition shipments.

Communist China struck a rather unique tone - while the Beijing government professed "resolute support" for India, Chinese Communist Party organs (such as its newspaper in Hong Kong) took a slightly different line, calling the invasion “a desperate attempt by Mr. Nehru to regain his sagging prestige among the Afro-Asian nations,” and ridiculing India for fighting "the world's tiniest imperialist country." It's worth noting that China and India would themselves fight a border war ten months later.

In the grand scheme of the early 1960s, for all the public concern noted by the states above, in non-Indian/non-Portuguese public opinion, the Goa Crisis was a relatively minor issue, especially compared with major Cold War-related crises that occurred before, during and immediately after, such as the Congo Crisis and the Cuban Missile Crisis.

These reactions are largely pulled from:

Keesing's Record of World Events, Volume 8, March, 1962 India, Portugal, Indian, Page 18659 available here

The NATO charter contains text that explicitly limits the scope of NATO collective defense to Europe and North America.

Article 5 - The Parties agree that an armed attack against one or more of them in Europe or North America shall be considered an attack against them all and consequently they agree that, if such an armed attack occurs, each of them, in exercise of the right of individual or collective self-defence recognised by Article 51 of the Charter of the United Nations, will assist the Party or Parties so attacked by taking forthwith, individually and in concert with the other Parties, such action as it deems necessary, including the use of armed force, to restore and maintain the security of the North Atlantic area.

Any such armed attack and all measures taken as a result thereof shall immediately be reported to the Security Council. Such measures shall be terminated when the Security Council has taken the measures necessary to restore and maintain international peace and security .

Article 6 - For the purpose of Article 5, an armed attack on one or more of the Parties is deemed to include an armed attack:

• on the territory of any of the Parties in Europe or North America, on the Algerian Departments of France, on the territory of Turkey or on the Islands under the jurisdiction of any of the Parties in the North Atlantic area north of the Tropic of Cancer;

• on the forces, vessels, or aircraft of any of the Parties, when in or over these territories or any other area in Europe in which occupation forces of any of the Parties were stationed on the date when the Treaty entered into force or the Mediterranean Sea or the North Atlantic area north of the Tropic of Cancer.

Therefore, the NATO alliance could not be invoked by Portugal, because Goa is outside of the scope of the NATO treaty.

That language was included in the NATO charter because the United States was eager to keep the alliance an arrangement to contain the Soviet Union. The US really wanted to avoid NATO turning into an alliance to uphold European colonial empires.

I am aware the USA wanted European countries to decolonize and Goa was a colony but India who was an ally of the USSR was the one who would annex it.

That's not accurate. India was not really an ally of the USSR.

Yes, India purchased arms from USSR, and sought ties to counter US-Pakistan ties in the 1950s. Yes, Nehru instituted socialist domestic economic policies. However, India was also a founding member of the Non Aligned Movement and publicly espoused friendly neutrality towards US, USSR and China (until late 1950s).

By 1961, the US and India had fairly strong relations, and US and USSR were competing with each other to provide developmental aid to India.

But, if we imagine a counter-factual where US, UK, France intervene and provide materiel support or troops in support of Portugal, such action could be expected to anger Nehru and inflame Indian nationalist feeling. That is, it might push India closer to USSR.

From a realist IR perspective, keeping good or neutral relations with the 8th largest economy, a country of 450 million people; far outweighed the importance of keeping Goa Portuguese.

NATO and the US were not happy at all that India invaded Goa, not because one of their allies was attacked, it's that they preferred a peaceful resolution to the situation. India had built much of their international reputation as a non-aligned state during the Cold War around peaceful resolutions to international disputes. The US attempted several efforts through the United Nations to arbitrate the issue of a European country continuing to occupy a foreign country to no avail. They wanted a resolution to the crisis, just not one that involved warfare.

Yes, the US was in favor of dismantling European colonial empires, not to replace them, but to prevent them from becoming hot spots of smaller wars that could potentially draw in NATO and the Warsaw Pact. This was not a new or unique stance as the US had already refused to offer more than nominal support for the French in Indo-China where communist support was already growing, and refused altogether to support the British and French in the Suez Crisis as it had global trade and power implications.

There was absolutely no benefit to supporting Portugal fight a war that their own military admitted they couldn't win on their own. It would be a diplomatic disaster. The "Free Democracies" of the world fighting a war to maintain an exploitative colony that was a relic of a time long past? Might as well just distribut Soviet propaganda yourself.

This might be a question in itself, in which case I'll post it as such but

India who was an ally of the USSR

I was always under the impression that India was non-aligned like Sweden. Were they Soviet allies in the 60s?

I had prepared a detailed report on it in college but that is lost. I still have a shorter presentation on the International reaction https://www.slideshare.net/RachitAjitsaria/international-reaction-on-goa-liberation

The top comment already answers the legal question of why NATO could not intercede. The reasoning of non-intervention have also been discussed. For the on ground situation, India was militarily weaker but still a significant domestic and diplomatic power. The war from the onset was one sided as Portugal didn't have a lot of local military. It tried to send reinforcements but met roadblocks. Egypt denied access to Suez Canal. Pakistan denied access to aerial passage. Portugal tried to have the UN Security Council intervene but Soviet Union vetoed the proposal. So the conflict was a short 36 hour affair and the powers to be were able to wait and condemn.

The reason for non proactive help of a prominent British ally and a NATO member by the western countries is that they were of the opinion that Portuguese should leave Goa. The situation did not escalate in a day. Nehru had been trying diplomatic means to resolve the situation and had asked USA to nudge Portugal to engage in talks. India had also tried blockade, essentially blocking off the smaller Portuguese enclaves of Dadra and Nagar Haveli. The broader western leaders perception could be summarized in the statement of British opposition leader "that the existence of Portuguese colonies on the Indian mainland had long been an anachronism and that Portugal should have abandoned them long since in pursuance of the example set by Britain and France."

Coming to the next part about the public reaction, USA did denounce India. Kennedy is reported to have said "You spend the last fifteen years preaching morality to us, and then you go ahead and act the way any normal country would behave ... People are saying, the preacher has been caught coming out of the brothel." The New York Times carried an article condemning India called "India, the Aggressor" ( https://www.nytimes.com/1961/12/19/archives/india-the-aggressor.html ). The Life magazine also carried an article against India and Nehru.

While, in broader post World War 2 history, the event is not given utmost importance, to any scholar the aftermath to the events provide a window to the times to come for partnerships and conflicts in the Indian subcontinent. The official Chinese Communist Party statement was in support of the military action and recognized Goa as an Indian territory. But the media portrayal in China was negative, calling it a stunt by Nehru to improve his electoral campaign and improve his diplomatic image in Asia-Africa by taking on the smallest imperial power. It is to be remembered that within a year China was to attack India, in a move that India and the rest of the world did not really anticipate.

Sources: https://history.state.gov/historicaldocuments/frus1961-63v23/d219 Letter from Pakistan PM to Kennedy https://history.state.gov/historicaldocuments/frus1961-63v23/d219 International Reactions to Indian Attack on Goa. - Soviet Veto of Western Cease-fire Resolution in security Council. https://web.stanford.edu/group/tomzgroup/pmwiki/uploads/1074-1962-03-KS-b-RCW.pdf LIFE 5 January 1962, Vol. 52, No. 1, ISSN 0024-3019, Published by Time Inc India and the United States: Estranged Democracies, 1941–1991 By Dennis Kux Published by DIANE Publishing, 1993