I answered a similar question here awhile back. The short version is:

• no, the Japanese did not think they were capable of invading the U.S., and did not assume they would need to do so to end the war. (In the event, the Japanese would barely have been able to sustain a garrison on Midway had that invasion succeeded; Hawaii was utterly beyond their reach, to say nothing of the West Coast.)

• The Japanese war aims were to grab control of the resource areas of the Southwest Pacific, particularly oil, rubber and other strategic resources in Borneo and Indonesia. The difficulty is that the Philippines, in 1941 a U.S. possession, are located between Japan and those resources, which themselves at the time belonged to European nations. So any attack on that area requires neutralizing US power in the Philippines.

• Given that strategic imperative, the Japanese figured that in any Pacific war that involved the US, the impulse of the American fleet would be to sortie immediately in the defense of the Philippines. Japanese doctrine was to meet that attack by attritting the US fleet as it moved across the central Pacific, by submarine warfare, airstrikes, and then coordinated torpedo assaults by destroyers and cruisers, before a final decisive battle somewhere near the Philippines that would force the Americans to sue for peace.

• The Japanese fleet studied carefully naval warfare in Europe, and were in particular impressed by the British attack on the Italian port of Taranto. Experimenting with torpedoes, the Japanese figured out how to get them to drop only minimally into the water when dropped from a plane (normally air-launched torpedoes would have buried themselves in the harbor bottom at Pearl) and decided that as an opening blow, removing the American fleet from Pearl Harbor would buy them extra time to form a defensive perimeter that would be needed to repulse the inevitable American counter-thrust.

• The Japanese were absolutely aware that they could not win a long war against the US; but, their military leadership that effectively ran the government was stuck in a war in China that it could not win without the resources of the southwest Pacific, and was facing a coup if it didn't win that war. They gambled on the US being forced to negotiate after a short war, and lost.

As far as the question of the oil tanks at Pearl Harbor, the idea that there was a "third strike" which was canceled is an untruth that was spread mainly by Mitsuo Fuchida. While in hindsight it seems clear that striking a blow against storage tanks would have been a good idea for the Japanese, in the event they had time for two raids that caught the Americans unprepared; further strikes would have used tired pilots and damaged plains and been against an island that was on high alert and eager to fight back.

Did Japan really think it was capable of invading the US? Were they hoping for a few land grabs and then a peace treaty? I'm trying to grasp what their end game was in the war. Didn't it turn into defensive war relatively quickly (battle of midway)?

So to dig into some of this.

No the Japanese senior leadership never contemplated an invasion of any sort of the mainland full stop. Even talks about invading Hawaii were considered little better than fever dream flights of fancy looked askance at. This was for a few reasons then.

1. Being that the islands were not self sufficient, considering the great numbers of civilians, and the needed garrison it would have taken a major commitment to keep the seized islands from starving. These being ships which would have limited resources to then carry back to the Home Islands as opposed to crucial Southern Resource Area. The Japanese Merchant Marine was already badly undersized for war in 1941 and every diversion not keeping resources and manufacturing flowing was a double hit, before any losses were thought about from enemy action.

2. A seizure of Oahu was not consistent with Japanese war strategy. it was a net drain on resources, far too distant from other bases to be mutually supporting like bases in the Mandates or South Pacific. It would be ripe for isolation by a US counterattack whereupon there was no easy way to relieve it except to come and fight a fleet battle on terms favorable to the US.

3. There just weren't the men. the IJA essentially determined how many units they were willing to commit to the war effort away from the Asian mainland. IJN and Joint estimates thought 3 division might be needed for Hawaii, which is probably an underestimate, however this represent almost half number of troops spared for the entirety of the drive South to begin with!

Many of these same concerns were raised when Yamamoto began his push for the Midway Plan, in particular that he might use army troops in a bait and switch to also reach out and seize Samoa or Fiji, once again pushing the perimeter outward. It was the urgency and shame of the Doolittle Raid that did much to bring them into line with his plan then.

For a more thorough examination of the topic I would look into Shattered Sword by Parshall and Tully who also express deep doubts about the ability of the IJN to even successfully take Midway as planned. Their website also examines the problems related to an invasion of Oahu in the context of a successful Battle of Midway but also play around with an expanded Pearl Harbor operation too: http://www.combinedfleet.com/pearlops.htm

I'm trying to grasp what their end game was in the war. Didn't it turn into defensive war relatively quickly (battle of midway)?

Jschool has aptly addressed this. The idea was basically to buy enough breathing room to build an outer shell hard enough o crack that even if it was the enemy would be vulnerable to annihilation or be brought to the table.

After Midway the IJN saw it as a serious loss but were not yet thinking in the terms of having lost all initiative. Certainly not when we look at how they fought the Guadalcanal Campaign, it would take another 6 months of bloody fighting in the Solomons to drive the point home.