This is a frequently asked question.

http://www.reddit.com/r/AskHistorians/wiki/asia#wiki_why_are_some_parts_of_southeast_asia_muslim.3F

In short, Malaysia and Indonesia were the midway point of the trade route between India and China. Muslim traders from India and the Middle East came to Malaysia and Indonesia and brought missionaries with them.

As for what religions were popular before, Hinduism and Buddhism were popular at one point (also spread by Indian traders). Before that, there was traditional Malay animism.

one of the most overlooked aspects of the early rise of islam was the cultural aspects of trading. whenever merchants of different cultures were dealing with each other, it made business run a lot smoother whenever you had similar religion, values, and business ethics. the original hadiths by muhammad actually say a lot about how to ethically conduct business. these business practices transcended physical geographical boundaries. so you see the phenomena of religion spreading through the trade routes motivated by international commerce.