How did the hindu population of south east asia decline in the face of islam, when it held on in india?

I can answer the second part of your question, namely why Hinduism held on in most parts of the Indian subcontinent despite long periods of Islamic rule. For the first half of your question, you can check out the posts by u/PangeranDipanagara who has already talked about the Islamization of Southeast Asia quite extensively. The following post of mine is essentially a repost of earlier posts I made on this subreddit.

The best way to understand why much of India did not convert to Islam is to look at the parts of the Indian subcontinent that did become Islamized, namely parts of the northwest like West Panjab and Sindh (in what is now Pakistan) and East Bengal (what is now Bangladesh), and juxtapose them with the rest of the Indian subcontinent. When we look at these two areas, one of the things we notice is that both the territories of modern-day Pakistan and modern-day Bangladesh were relatively marginal and economically underdeveloped at the time of the Islamic conquests. Neither area had a substantial sedentary and urban civilization that could support powerful empires. All of the great North Indian empires were centered in the rich Gangetic plain, which included great cities like Pataliputra (Patna) and Kanyakubja (Kannauj), while Pakistan and Bangladesh had no urban centers of comparable size and influence until the rise of Lahore and Dhaka in much later times. Whatever states existed in Pakistan and Bangladesh were localized and restricted in scope, while polities based in the rich Gangetic plains (like those of the Mauryas and Guptas) had a far wider influence.

Because of their marginal nature, both Pakistan and Bangladesh could be characterized as "frontier" regions, where Muslims could (and did) make a big impact on the native population. Pre-Islamic dynasties in Bengal like the Palas and Senas had their centers of power in western Bengal, such as at Gauda/Lakhnauti and Nabadwipa, and not in the backwaters of eastern Bengal. It was under Islamic rule that the eastern parts of Bengal were intensely settled and cultivated, and because of the supervising role that Muslims played in this process, they became the local elites of the newly-civilized society in eastern Bengal and were able to establish their influence at a grassroots level, absorbing many Bengalis into the Islamic fold. Western Bengal, by contrast, already had a well-established social and economic order that was long under the domination of a deeply conservative Hindu aristocracy, and so Islamic influence in western Bengal was much weaker at the local level. A similar dynamic also explains why frontier regions in modern-day Pakistan became thoroughly Islamized while the long-settled and highly-stratified regions of UP and Bihar remained strong bastions of Hinduism. A large portion of the local elites and landlords in Pakistan were Muslims (including many foreigners like Pathans/Afghans), and they encouraged the settling of their landholdings by formerly nomadic or semi-nomadic Hindu tribes, who subsequently became Islamized. In contrast, Hindu notables in UP and Bihar retained local influence and authority, and so Hindu social structures in those regions were preserved to a much greater degree.

Economics Indonesia specifically began its conversion because so many of the spices that drove econonics. Many spices were exclusive to only a few specific islands that were well guarded. Most of these were in the Maluku Islands. Nutmeg for example was only grown on 3 islands. Because Traders were moving from the middle east the traders brought islam to a very open minded population that very naturally converted.

Naturally this trade network consolidated into a Kingdom Srivijaya, Hindu and Buddhist. But information on this kingdom is limited.

As for the mainland Hinduism was folded into the traditions of Thai and Khamenian culture. Even today you can see hundreds of shrines to hindu gods and Thailand carries its own version of the Ramayana. And local people regularly switch between temple types and spritualism. If you take this modern open approach to religion you can see how receptive the population was to islam.

Believe Systems harden with the arrival of European Conflict (Portugeuse v Dutch.)

-Anthony Reid, History of Early Modern SEA And personnal travels