Was it that unlike the East Indies it had no luxury resources such as Spices or did they just not have the resources to establish a colony like the British did?
The Dutch East India Company (VOC), which was active in the area until 1800 was a trading company. Like its British counterpart it did control an ever increasing amount of territory, but it all served trade in the end, by ensuring supplies or allowing them to hold on to monopolies (for example, they controlled the only island where mace grew, and went to great lengths to keep it so).
At the time, it was thought that there must be a vast southern continent, to balance out the continents of the Northern hemisphere, Terra Australis Incognita (the unknown southland) an idea that goes back to the ancient Greeks. There were high hopes that this continent would provide ample opportunities for trade. So, several expeditions were sent by both the East and West India companies (after all, who knew how far west the Southland extended). As a result, New Zealand, Tasmania and several pacific islands (Easter Island among them) were discovered, but no vast, rich continent. There were also three expeditions to the western coast of Australia to rescue shipwrecked sailors (not to mention the cold and silver that had been on board). As a result, the northern, western and southern coastlines were fairly well mapped.*
Sure, they could have stayed in Australia and started, say, sugar plantations. But why would they? They already had plantations much closer to their base of operations, and with ample labour available (not enough aboriginals around for a plantation economy). Turning Australia into a profitable colony would have meant investing a lot of resources and money, which could be much better spent elsewhere.
EDIT: Interstingly, i just cam across this book, in the appendix (p220) it says that two mutineers (survivors of one of the aforementioned shipwrecks) were put ashore on the Australian mainland. So i guess the Dutch did colonise the place before the English ;p
*Note that Tasmania is still called Van Diemen's land, after Tasman's employer, two other features are named after Van Diemen's wife Maria.
Actually it was the portuguese who discovered AUS
