There were plans made for such an attack. In 1941 and early 1942, Australia posed a major threat to Japanese plans. Troops and ships escaping from the Dutch East Indies could flow southwards to bases in Western Australia. Aircraft based in the Northern Territory could interfere with Japanese movements towards Papua New Guinea, and of course, Australia could provide a base for an Allied counterattack.

The Imperial Japanese Navy had a doctrine that sought to keep the Allies consistently on the defensive. To this end, they planned attacks into the Indian Ocean, and towards Hawaii with Midway as an objective. During this time, they also drew up plans for an attack on Australia. Plans for this were in place, and being pushed by the Navy's general staff as early as December 1941. These plans sought to capture several strategic positions on the northern and north-Eastern coasts of Australia. As it was considered difficult for Australian troops to reach these locations, it was thought that these locations could be captured with a minimum of shipping and expenditure of troops and materiel. (The lack of strategic mobility on land came from a lack of rail lines and roads to much of the north coast) Navy estimates suggested that between 45,000 and 60,000 men would be needed to capture their chosen targets.

However, the Army had a different outlook. Their commitments in China, Manchuria and South-East Asia made it difficult to find the men needed for further expansion and offensives. Instead, they preferred a defensive strategy, which they felt more fitting with Japan's capabilities. They believed that an invasion of Australia would need to capture more of the eastern coast, which was more heavily defended. Their estimates for the numbers of troops needed for such an invasion were closer to 150,000 to 200,000 men. This number was impossible to find without heavily weakening the Japanese force in Manchuria, which the Army was always unwilling to do. In addition, they calculated that 1.5 - 2 million tons of shipping would be needed to support these troops, an amount of tonnage that could not be found without destroying the foundations of the Japanese war economy. Realising the need for offensive operations against Australia, they instead proposed a strategy based around cutting its lines of communication with the USA, by taking New Caledonia, Samoa and Fiji. They were supported in this project by Japanese PM Hideki Tojo, who was always opposed to an invasion of Australia.

These two sides fought it out in bitter bureaucratic warfare through February and early March 1942. An Imperial Liaison Conference on the 7th March made only oblique references to an attack on Australia, with its report making statements that could be referenced as supporting an attack, but concluding that Japan should be "building a political and military structure capable of withstanding a prolonged war", phrasing that implies caution about opening new fronts. However, on the 13th, a new strategic plan was reported to the Emperor. This new plan relegated an attack on Australia to "a future option to demonstrate positive warfare", with the qualification that the situation with the Soviet Union and China allowed it. Instead, the Army's plan, modified by the capture of Port Moresby, was to be followed.

Unfortunately for Japan, they wouldn't get a chance to put these plans into action. The Battle of the Coral Sea scuppered the attack on Port Moresby. Midway caused a postponement of the attacks into the South-West Pacific, and allowed the US to intervene with the preliminary attacks into the Solomon Islands.

References:

Brown, G., Anderson, D., Invasion 1942? Australia and the Japanese Threat, https://www.aph.gov.au/binaries/library/pubs/bp/1992/92bp06.pdf

Frei, H. P., Japan's Southward Advance and Australia