The Japanese Navy considered invading Northern Australia, which was largely unpopulated and poorly connected to the rest of the country. The Army were generally opposed to such an operation. Had such an invasion gone ahead, they wanted to extend it to include the more populated eastern coast. However, these plans had to be scrapped, as the required troop numbers were considered impossible to spare, ship and support. These plans also lacked political support, with Tojo being strongly against them.

Instead, moves were taken to cut off Australia and New Zealand from the main convoy routes to the US. Their landings in the Solomon Islands were intended to establish air bases to cut these routes. Invasions of Samoa and Fiji, which also lay astride these routes, were also planned, as well as an invasion of New Caledonia. However, Coral Sea and Midway resulted in the loss of the carrier support necessary and led to the cancellation of these invasions.

While the Japanese did attack Australian bases at Darwin and Broome from the air, this was to prevent them from being used to support Allied advances into Indonesia, rather than as a prelude to an invasion. Similarly, the Japanese attempts to capture Port Moresby were not aimed at using this base as a springboard for an invasion, but rather to use it to control the seas around New Guinea.

Sources:

Toland, J. , Rising Sun: The Decline and Fall of the Japanese Empire 1936-1945, Pen and Sword Military, 2005

Brown, G., Anderson, D., Invasion 1942? Australia and the Japanese Threat, https://www.aph.gov.au/binaries/library/pubs/bp/1992/92bp06.pdf

the Japanese didn't want to invade Australia, but more isolate it as it would have been unpractical to take and hold the country. the Australians were very worried about it due to the Japanese taking control of much of the south pacific during the onset of American involvement in WW II. here is a link to that specific question A large naval battle, the Battle of the Coral Sea was fought very near to Australia, the Fall of Singapore, and the invasion of the Philippines all took place near enough to Australia to cause legitimate concerns.