Aside from some Naval harassment of the coast, there was never a serious Japanese invasion plan for North America. The Imperial Navy's grand strategy for the war was to bring an armistice settlement after decisively defeating the U.S. Navy's Battleship fleet in waters close to Japan, much in the way that Tsushima strait decided (according to the IJN) the Russo-Japanese war. Surprisingly, the Japanese army was relatively uninvolved in the war planning against the western Allies, due to intense inter-service rivalry.

Of course, fighting a very large war in China and supplying an invasion force on the other side of the Pacific was so far beyond Japanese manpower, shipping and logistical resources that it would have been absurd to plan. Even capturing Midway Island might have required more shipping that Japan could produce, let alone Hawaii, and North America was completely unrealistic.

Edit: Source here is Evans and Pettie's "Kaigun"