There certainly was a strong belief in a large undiscovered land mass in the unexplored portion of the globe around this time, but it was thought to lie further to the south.

I covered the basics of what was believed about this mysterious continent, which appeared on charts of globes for centuries, in my book Batavia's Graveyard (2001):

‘Terra Australis Incognita’, they called it: ‘the unknown South-Land’. Even in 1629, its very existence was based more on supposition than on fact. Early geographers, such as the Græco-Egyptian Ptolemy, writing in 140AD, had imagined a world divided into four gigantic continents. Europe, and what was known of Africa and Asia, was believed to occupy the north-east portion of the globe. This massive land mass seemed to require a counter-balance. From the earliest days, therefore, world maps showed a giant continent south of the equator, girdling the earth and in many cases joining South America and Africa to China.

As the Portuguese and Spaniards pressed southwards in the fifteenth and sixteenth centuries, it gradually became apparent that the South-Land could not be as big as it had been supposed. Ships rounded the Cape of Good Hope and Cape Horn without sighting it, and sailed north-west across the Pacific and east through the Indian Ocean without finding any trace of the mysterious continent. By the time the VOC was founded, almost the only place left to look was the great blank that still lay south of the Indies and west of the Americas. 

Contemporary globes and maps continued to indicate the presence of Terra Australis in this area. Over the years, elements of fantasy had crept into descriptions of the South-Land, and in the sixteenth century faulty interpretation of the works of Marco Polo led to the addition of three non-existent provinces to maps of the southern continent. The most important of the three was Beach, which appeared on many charts with the alluring label provincia aurifera, ‘gold-bearing land’; sailors often referred to the whole South-Land by this name. The other imaginary provinces were Maletur (scatens aromatibus, a region overflowing with spices) and Lucach, which was said as late as 1601 to have received an embassy from Java. The existence of these provinces was an article of faith for most Europeans; in 1545 the Spaniards had actually appointed a governor of Beach – a certain Pedro Sancho de la Hoz, who was one of the conquistadors of Chile. Even the more pragmatic Dutch did not entirely disbelieve, for their ships had occasionally stumbled unexpectedly across a coast that they believed must be part of Terra Australis.

In the first years of the VOC, the Company’s sailors had largely kept to the sea lanes established by the Portuguese. From the Cape of Good Hope, these ran north along the African coast to Madagascar, and thence north east across the Indian Ocean to the Indies. There were, however, significant problems with this route. The heat was frequently unbearable, the Portuguese unfriendly, and there were numerous shoals and shallows to negotiate along the way. Furthermore, once north of the Cape, contrary winds and currents made the voyage extremely slow; journeys of up to 16 months were not uncommon. There were also frequent hurricanes, which caused the loss of many ships. The Dutch persisted with the Portuguese route, unsatisfactory as it clearly was, only because they knew of no alternative.

Then, in 1610, a senior VOC official named Henrik Brouwer discovered an alternate passage far to the south of the established sea lanes. Heading south rather than north from the Cape of Good Hope until he reached the northern limits of the Roaring Forties, he found a belt of strong, consistent westerlies that hurried his ships towards the Indies. When Brouwer estimated that he had reached the longitude of the Sunda Strait, which divides Java and Sumatra, he had his ships turn north and reached the port of Bantam only five months and 24 days after leaving the United Provinces. He had cut about 2,000 miles from the journey, outflanked the Portuguese, more than halved the time taken to complete the outward voyage, and arrived in Java with a healthy crew to boot.

The VOC's directors were suitably impressed. Faster voyages meant increased profits, and from 1616 all Dutch ships were enjoined to follow the ‘fairway’ Brouwer had discovered. So long as the VOC’s skippers kept an accurate reckoning of their position, it was undoubtedly a far superior route. But the strong winds and fast currents of the Southern Ocean made it all to easy for to under-estimate how far east a ship had sailed. When this occurred, the vessel would miss the turn to the north and find herself sailing dangerously close to the barren coast of Western Australia.

There were several near-disasters. In 1616 the Dutch East Indiaman Eendracht unexpectedly encountered the South-Land after an unusually fast passage from the Cape, and sailed north along the coast for a few hundred miles. The charts her officers drew were incorporated into the VOC’s rutters, which henceforth indicated the existence of a small portion of the Australian littoral, called Eendrachtsland; but it was by no means certain at the time whether this new coast was the South-Land or some smaller island. In any case, communication with Europe was so slow that news of the discovery took a long time to reach the ears of many skippers and when, two years later, another ship – the Zeewolf – chanced upon what was almost certainly the North West Cape, her skipper was considerably alarmed ‘as we have never heard of this discovery, and the chart shows nothing but open ocean at this place.’

Sources

De Jode’s atlas, Speculum Orbis Terrae, notes: ‘This region is even today almost unknown, because after the first and second voyages all have avoided sailing thither, so that it is doubtful until even today whether it is a continent or an island. The sailors call this region New Guinea, because its coasts, state and condition are similar in many respects to the African Guinea... After this region the huge Australian land follows which – as soon as it is once known – will represent a fifth continent, so vast and immense is it deemed...’ See also Günter Schilder, Australia Unveiled: the Share of Dutch Navigators in  the Discovery of Australia (Amsterdam: Theatrum Orbis Terrarum, 1976)  pp.268-9; Miriam Estensen, Discovery: the Quest for the Great South Land (Sydney: Allen & Unwin, 1998) pp.5-9. The name ‘Terra Australis Incognita’ appears on Henricus Hondius’s famous world map of 1630 (ibid pp.320-1). Abraham Ortelius’s Types Orbis Terrarum (ca.1600) gives ‘Terra Australia Nondum Cognita’ (ibid pp.266-7), and there were several other variants.