Yes, they thought they could defeat the USA. They didn't expect to invade and occupy the continental US, but planned to present the US with a fait accompli, that would allow Japan to secure a negotiated peace that was highly in their favour.

Japan had two major war aims - secure the oil and natural resources they needed to continue their war in China and to prevent the USA from intervening. The Philippines, an American colony at the time, was perfectly placed to interdict the sea lanes from the oil fields of Borneo and Sumatra. The American fleet had to be destroyed immediately, to allow time for a defensive perimeter to be set up to blunt and destroy the inevitable counterattack. The Japanese assumed that with Indonesia and the Philippines under their control, and the American counterattack having been defeated, the US would be forced to negotiate. They underestimated the ability of the US to keep fighting in such a situation.

I'd recommend John Toland's work 'The Rising Sun' for more details on how and why Japan planned to fight the Pacific War, as well as for details on the fighting.