This is a complicated question, and a pretty tragic one, as institutional inertia and interservice rivalry interfered constructively to push Japan down a road not in its own best interests.

The Imperial Japanese Navy has a long and complex history, both accepting aid and training from and also considering enemies multiple different Western nations. French jeune ecole theory, British advisors, Mahanian decisive battle strategy -- it was an organization eager to learn the state of the art of naval warfare from around the world.

But it was also locked in a fierce rivalry with the Imperial Japanese Army for resources and prestige, partly because of the way the Japanese command structure was set up. This meant that to justify their own increasing requests for money and resources, the Navy needed an enemy that realistically threatened Japan.

Hypothetical Enemies

Naval officer Satou Tetsutarou was chosen by the leadership of the navy to write a treatise in 1902 titled Teikoku Kokubou Ron/On Imperial Defense. It caused huge controversy as it said that the only real defense an island nation could depend on was its navy, reducing the army to a secondary role. He went on to write several more treatises, saying essentially that the only way to defend their island was to have a navy capable of blue water offensive operations, and to destroy the enemy at home before they could get near the Home Islands.

Satou also played a key role in identifying the "hypothetical enemy" which would serve as the yardstick against which the Navy's needs would be measured, but unlike other nations who chose their hypothetical enemies based on diplomatic disposition or threat -- think Britain and Germany's naval arms race -- he advocated for choosing a potential enemy based on their capability to harm Japan, whether they indicated they had any intention of doing so or not.

He argued that the United States was the most dangerous hypothetical enemy, and should be chosen as the focus of the Navy. Furthermore, as America would have to divide its forces between the Pacific and the Atlantic, it was decided that Japan's navy should aim to be no less than 70% the strength of the US Navy in order to reasonably defeat it.

Painting into a Corner

Decades of political infighting followed, with the Army and the Navy arguing before the Diet and the throne as to who should be given priority. Various factors continued to push the Navy's leadership further into their imaginary rivalry with the United States; anti-Japanese racial sentiment on the West Coast, the rapid growth of the American Navy, the Washington Naval Treaty limitations. This constant in-fighting and grappling for power kept the Navy's attention focused on short-term concerns, rather than long-term ones.

The advent of oil-powered warships as an advancement over coal-fired ones pulled the Navy's vision even closer in, as Japan had briefly made itself self-sufficient in fuel with local coal, but had nothing close to the amount of domestic oil production to power the new ships the Navy demanded be built to keep its hypothetical pace with America. The Japanese Navy began stockpiling huge amounts of oil, keeping exactly how much a strict secret even from other government organizations. From this point forward, oil supplies would take an enormous role in Japanese naval strategic decision-making. In contrast to the Army, which had pushed for a "northward advance" against Russia and China, the Navy advocated for a "southward advance" into the Indies and its rich resources.

1940-'41

In July, the Army complicated the Navy's position by abruptly announcing they were also interested in a southward advance, focused on taking French Indochina. They proposed that while they might go to war with the colonial governments of the Indies or even Britain, war with America could be avoided.

To the Navy's leadership, this amounted to a dagger thrust at the heart of the navy. Eliminating the US from the conflict meant that the Navy's main reason for existing for decades would be eliminated, and the Army would win all the glory of the southward expanse. This briefly made the Navy change its mind and advocate for a diplomatic attempt to gain the resources they needed, refusing to sign on to the Army's southward advance.

America's declaration that it would meet the use of force in the Pacific with strict embargoes drove the Navy back into the arms of the Army. It began preparing for what it saw as inevitable conflict with the US Navy, with the full knowledge that they had the resources to fight for a year at most. But it was too focused on its own short-term, internal goals to be swayed much by long term concerns. Because of this, the Navy reluctantly agreed with the Army's advice to sign the Tripartite Pact with Italy and Germany, in exchange for the Army agreeing to a reallocation of more resources to the Navy.

By the Spring of 1941, the Navy had decided that war with the US was inevitable, and it should look to control the timing to its best possible advantage. No more waiting could be allowed. Admiral Nagano is quoted as having said, "The Navy is consuming 400 tons of oil an hour. The situation is urgent. We want it decided one way or the other quickly."

The Navy had essentially caught itself in a web of circular reasoning of its own construction, driving it into war by necessity of having built itself up to fight that same war.

Yamamoto

Isoroku Yamamoto, though largely opposed to war with the US and UK, was commander of the Combined Fleet and thus had a responsibility to execute the orders he was given. After successful naval aviation exercises in the spring of '40, he began to realize the potential of a long range air strike on the USN's naval assets to open the war. He submitted these proposals in a memorandum to the Navy Minister in January of '41. The Navy General Staff was not fully on-board with Yamamoto's plan until as late as October of 1941, scant weeks before the attack was depart from Japan for Hawaii.

Conclusions

This is greatly abridged and summarized, of course, leaving out a lot of important personalities and details, but I think gets across the basic idea that the Navy, and Japan by extension, trapped itself in a box of its own making from which they decided war was the only escape. This wasn't driven wholly by the best strategic decisions for the country itself, but by strategic decisions for the Navy versus the Army.

Making decisions by a process like this turned out to be exactly as bad an idea as it sounds in hindsight.

Source:

Evans, David C., and Mark R. Peattie. Kaigun: Strategy, Tactics, and Technology in the Imperial Japanese Navy, 1887-1941. Annapolis: Naval Institute Press, 1997.

u/KDY_ISD has written an excellent summary of why the Japanese started the war, but did not really get into the Pearl Harbor side of your question and barely touched on America.

First off, we need a map. For all the reasons u/KDY_ISD has stated, the Japanese decided to take the Dutch East Indies, an area rich in oil and rubber among other resources. This would naturally lead to war with Great Britain, which at this point was the home of the Dutch government in exile. Japan (rightly) expected they could easily defeat their military forces in the area.

However, note the location of the Philippines on this map. They sit directly between Japan and her new Southern Resource Area, and posed a major threat to the Japanese supply lines. At this point the islands were owned by America, and they had several major Army, Army Air Corps, and Navy bases in the area. If for any reason the US entered the war, they could shut down the Japanese supply lines in a moment, which is exactly what happened when the US took the islands in 1944-1945.

This was a major risk, and the Japanese command structure debated the situation for some time. Admiral Osami Nagano, head of the Naval General Staff, argued that the US would find it difficult to go to war if Japan only attacked British and Dutch colonies. However, I found a November 1941 memo to Roosevelt in the National Archives (second pdf, page 73) where General Marshall and Admiral Stark is quite clear they expect war to come eventually, even setting up lines in Thailand Japan should not cross, and that the key point was buying time to reinforce the US positions.

Admiral Isoroku Yamamoto, commander in chief of the Combined Fleet, won the day. He threatened the Naval General Staff to accept his plans and go to war with the United States by attacking the Philippines and Pearl Harbor or he and his entire staff would resign. The Naval General Staff caved.

It is important to understand Yamamoto's reasoning, as this was arguably his only sound strategic decision of the war. Again, look at that map. To take all of these islands, Japan needed to devote most of her Navy to the job. For example, Japan had 18 heavy cruisers, 14 large and 4 small, in 1941: sixteen of them were assigned to the various invasion and covering forces. They had 68 modern destroyers in three broad capability groups: 47 were with these invasions. Two of their ten capital ships. Most of their light cruisers, old destroyers, and all their torpedo boats and minesweepers (for the Japanese these doubled as escorts and were unusually powerful). The carrier Ryūjō and all available seaplane carriers (a major emphasis in the Japanese Navy). This was an absolutely massive force, and if you want to see the number of ships involved Dr. Leo Niehorster has an excellent breakdown. Most of the ships not involved in the invasions were with the carrier force attacking Pearl (two capital ships, two heavy cruisers, one of the better light cruisers, and nine of the most modern destroyers with another two detached to shell Midway), others were in the Home Islands as a "Quick" Reaction Force (six capital ships, two carriers, two torpedo cruisers, ten modern destroyers).

In December 1941, Japan had a navy roughly comparable to the British, American, and Dutch forces in the Pacific. Each force had their strengths and weaknesses in certain categories: Japan had the edge in carriers and the Allies in light cruisers. The vast majority of the Allied forces flew the US flag, most in the Pacific Fleet based in Pearl Harbor. The plan was for these forces to sail west in response to a Japanese attack on the Philippines, and if that happened Japan would have to denude their invasion forces to counter the attack. That is a recipe for failure, so Yamamoto and his staff planned the attack on Pearl Harbor to keep the US fleet from sailing west. Based on Taranto, where the Italians repaired most of their ships in seven months, Yamamoto expected the US to begin sailing west in six months, just enough time to finish all the conquests in the Western Pacific. That was the goal of the Pearl Harbor attack, not completely destroying the US fleet, not forcing the US to capitulate, just keeping the US from interfering for six months.

As an aside, the United Stated knew that, if Japan attacked, they would start the war with a surprise attack, likely on the Philippines. Therefore, the surface forces in the Asiatic Fleet were all old and expendable (one heavy cruiser, one old light cruiser, 13 old destroyers, and several auxiliaries and minor combatants), though the submarine forces were extremely strong (29 submarines, including 23 of the most modern in service). When the various war warnings came out in the weeks before, most of the surface ships were moved south and out of harm's way, but the submarines continued operating out of the Manila area until the Japanese were just outside the city, and even after that made several supply runs to Corregidor.