A series of Dutch explorers mapped parts of the Australian coastline (and Tasmania) through the 17th century. William Janszoon was the first, in 1606. At least some of them (e.g. Abel Tasman) were 'continent hunting'; Tasman is credited for christening the continent 'New Holland' in 1644.

What we were taught in school in Australia was that the Dutch weren't that interested in new Holland, because (Tasmania aside) what they visited and mapped was seen as barren, infertile land. Tasman only stayed briefly in what is now Tasmania even though it probably appeared more 'fertile' to the mind of a European; he was on an expedition to a find a continent, not more islands.

I am aware of at least one Dutch East India Company proposal to establish a colony in what is now South Australia in the mid-18th century. This was ultimately rejected on economic grounds. Also consider the comparatively much greater economic opportunities the Dutch already controlled in the Malay Archipelago in this period.

The Dutch would've additionally been aware of the trepang industries along the north coast of Australia, which began at least as early as 1720. However, the market for trepang was largely in China and probably of little interest to them in comparison to the spice trade.

Some would suggest that the Dutch might've thought differently about the continent had they explored the southeast (i.e. what's now Victoria and NSW) rather than the north and the west.

British interest developed slightly later, right around the turn of the 18th Century. William Dampier, a British pirate/privateer landed on the west coast in 1699. Dampier's visit is credited as a catalyst for British efforts to explore New Holland.

And here's the bit you probably already know...the British Crown decided to establish a penal colony in Botany Bay in the 1780s. This came at the recommendation of the botanist Joseph Banks and American-born sailor and diplomat James Magra, both of whom had been on the Endeavour with James Cook in 1770.

I can't speak much to French interest in the continent myself, beyond knowing the names of a few French voyages in this part of the world during this period.