Short Answer:

"Secret instructions" were a common bit of naval and diplomatic command logistics at the time. Naval instructions had the interesting problem that they might be read as much as a year after the date that they were written-- those giving the orders often had a variety of highly provisional future scenarios in which they might be read; keeping important intentions secret would be one goal, maintaining a legal presumption of neutrality would be another.

Discussion:

It's quite a challenge to write an order to be executed at some unspecified future date, particularly at a time when European wars were common, but often resolved as well. Sealed orders -- of which the Captain sometimes knew the gist, but sometimes did not-- reflected the contingency of the situation.

In the case of Cook, though, the reason for the secret orders was international political intrigue-- he had an ostensible scientific mission, and he could honestly maintain that that was the only mission he was aware of. It was only when he reached the region that the second set of orders would become known to him; thus any prior denials by him -- had he been inquired of-- would have been honest.

Cook's secret orders indicate a classic exercise in deception of Britain's perceived enemies. The Admiralty's instructions to Cook, the Commander of the Endeavor show that his ostensible reason for the voyage, the observation of the transit of Venus at Tahiti was only a preliminary event, a cover for the real purpose of the voyage which was to discover and annexe valuable posts on the east coast of what was then New Holland

At this moment, there was a rush to discover what people imagined was "the great Southern Continent", one which at the time was held out to be of great wealth and importance

A Frenchman, Kerguelen, had been searching for that elusive continent; and Dalrymple, Cook's bitter enemy, now gave out that it contained over 50,000,000 of people, and must be far wealthier than our American colonies.

I should note here that the aim of Cook's mission was not "to find Australia" -- it was to find a Southern continent, and Australia when found was considered a bit of a disappointment-- exploration continued, with an expectation that another more prosperous land was out there, somewhere.

Because of the legitimacy of European land claims rested heavily on the "Doctrine of Discovery", there was an imperative to conceal from rivals (in this case the French and the Dutch) just where one might think such new land might be. This is also the imperative behind the careful surveys done-- as evidence "I was in this harbor first, and surveyed the coast" -- as Cook does with great care.

Secret orders have been used at other times when hostilities are contemplated and a nation does not want to tarnish a serving military officer as a spy. You can see this dynamic in then-Assistant Secretary of the Navy Franklin Roosevelt's instructions to Admiral William Sims; he's sent to Europe while the US is still neutral, but war is contemplated

Following his testimony on Jutland in Congress, Sims received orders to the Naval War College. In February 1917 he assumed duty as the President of the College. Sims then received secret orders to sail for London with verbal authorization to assume rank as a rear admiral on 21 March. Concurrently, Navy Secretary Daniels and CNO Benson directed Sims to act as the Navy Department liaison to the Admiralty in London. The United States declared war on Germany while Sims was at sea in April.

Sources:

Rose, J. Holland. “Captain Cook and the Founding of British Power in the Pacific.” The Geographical Journal, vol. 73, no. 2, 1929, pp. 102–122.

Bennett, Bruce. “Politics and Spying: Representations of Pre- and Early Australia.” Antipodes, vol. 22, no. 1, 2008, pp. 17–22.

Stanbury, Myra. “LOUIS ALENO DE SAINT ALOÜARN (1738–1772) A FORGOTTEN 18th-CENTURY FRENCH EXPLORER.” The Great Circle, vol. 39, no. 2, 2017, pp. 25–55.

Kohnen, David, et al. “THE U.S. NAVY WON THE BATTLE OF JUTLAND.” Naval War College Review, vol. 69, no. 4, 2016, pp. 122–145.