Admiral Nagumo was commander of the Carrier strike force, Yamamoto was on his own battleship, and while he was responsible for planning the attack, Nagumo was responsible for the Carriers, and thus would have had a good amount of freedom to do what he felt necessary. The point of the Carriers was not to sink the US fleet, rather they were there because Yamamoto was convinced that Midway Island could be neutralized by a carrier strike force and then seized by a small landing force. The idea was that by capturing Midway, the Japanese could use the airbase there to help support the Japanese fleet in the expected counterattack that the Americans were sure to launch. It was felt that it would only take one round of bombing to subdue Midway, when Midway continued to hold out after the initial wave of bombing Nagumo was put in an awkward position. He was forced to choose between launching another wave of bombers to hit Midway (his primary objective) or to keep some in reserve for the US fleet. He choose to bomb Midway, we know it was a mistake today, but Nagumo had no way of knowing what would happen. He was forced to make a tough decision and he made one.

Its also important to understand just how much of a role intelligence played in the battle of Midway. The Americans had broken almost of the Japanese top codes, where as Japan had broken almost none of the American codes(save for some non important ones). The Japanese were also unaware that the Americans would be able to send as many carriers as they did, and the Japanese assumed it would take too long for the American carriers to reach Midway. The Japanese and Yamamoto assumed that the damage done to the Yorktown at the battle of the Coral Sea, and the attacks against the Aleutian Islands in Alaska would distract the Americans and allow the Japanese to seize Midway Island relatively easily. Unbeknownst to the Japanese, the Americans knew that Midway was the primary target and had sent three carriers, including the repaired Yorktown, to attack the Japanese at Midway.

Here is a report from the Office of Naval Intelligence that is an interesting read on the battle of Midway.

http://ibiblio.org/hyperwar/USN/USN-CN-Midway/USN-CN-Midway-12.html

Edit: edited for clarity.

Before I answer your question, I would just like to mention that as Tully mentions in Shattered Sword, the rearming of the planes with torpedoes vs. bombs being the decisive moment is a false conclusion. Simply put, the Kido Butai was in no condition to be launching a strike at any point during the time the US dive bombers attacked.

It didn't matter if the reserve planes were on board the carriers or not. The problem was that after beating off the American torpedo plane attacks, the flight decks were crammed full of planes from the carrier combat air patrol that needed to be refueled and rearmed, along with returning planes from the strike on Midway. This not only made it difficult for any strike to be launched, but also left the combat air patrol above the carriers in a much weakened state. And as dive bombers attack from the vertical plane, rather than the horizontal one, surface-to-air ackack fire has a much more difficult time either destroying planes or forcing them to break off their attack runs. Just look at the battle of Santa Cruz, where despite protecting the carriers in a ring formation, the surface assets were unable to prevent US dive bombers from crippling the carriers Shoukaku and Zuihou.

As for why he ordered his reserve planes to prepare to attack, it was simply because the commander of the first strike on Midway reported his observed results and recommended a second strike prior to sending in the "bombardment force." Do recall that the Japanese remembered losing the destroyers Hayate and Kisaragi at Wake to land-based emplacements and land-based aircraft, respectively. They wanted to thoroughly smash the land-based defenses prior to committing naval assets to an area of vulnerability, especially as the bombardment force in question-the heavy cruisers Mogami, Mikuma, Suzuya, and Kumano-were equipped entirely with Sanshikidan (Type 3) shells, which were useful for setting things like airfields on fire (and, theoretically, for anti-aircraft purposes) but not as good for penetrating armor.

Also, you have to keep in mind that his main sources of intelligence at the time were a submarine screening force well in advance of the fleet as well as the two heavy cruisers Tone and Chikuma. Tone and Chikuma were interestingly designed cruisers with all their guns mounted fore in order to make room for a large amount of seaplane capacity in the rear of the ship. They would deploy their floatplanes to scout for enemy carrier formations, and relay the signal back to the fleet.

The problem was that, depending on whether you take the version from "Miracle at Midway" or "Shattered Sword," either Tone's plane or Chikuma's plane screwed up and either missed the US fleet or messed up the reporting, leaving a gap in their search pattern that wasn't uncovered until later. And the subs completely missed the carriers.

As a result, Nagumo suspected nothing at the time he made the decision to rearm planes (not that it would matter), because he assumed that, as the Midway planning had predicted, and as his scouts had not sighted a US fleet, the US carriers were not in the area. He thus did not think he had to worry about an enemy carrier threat until a belated report from the cruisers came in, by which time it was already way too late.

Nagumo was in a bit of a tough spot, because his orders also specified that Midway was the primary target, and that the American carriers should take several days to arrive from Hawaii anyway. Since his scout planes had already completed their searches and failed to find the American fleet, Nagumo elected to prioritize reducing Midway.

So, while Nagumo did disregard instructions to keep a reserve anti-ship strike force, he wasn't wandering completely off into left field but operating in reasonable pursuit of objectives clearly defined by the battle plan. Nagumo, as you probably know, was a pretty by the book officer.

You may want to compare to this to Nagumo ordering radio silence broken earlier in the voyage to Midway to make a fleet course change in heavy fog. In the immediate Japanese post-mortem of the battle, that was what was considered Nagumo's worst mistake - they assumed the Americans detected it, not actually the case, and that that was how operational security was blown.

especially in an organization like the IJN?

I'm curious what you mean by this; the IJN officer corps was not exactly a model of military discipline.

Sources for the above: Miracle at Midway, Shattered Sword

EDIT: removed tangent