First, a disclaimer: this question is predicated on Wikipedia's article on the Battle of Midway, and I would not be surprised if answers to this question turn out to be "the wiki article is shit".
That said, my impression of the battle is that the USN escaped with a victory due to a number of lucky breaks - including the effectiveness of dive bombers (as another answer here pointed out, they were "unreasonably" effective at Midway), the fortuitous arrival and location of the Enterprise and Yorktown squadrons (VB-3, VB-6, and VS-6) relative to VT-3 (which drew the opposing CAP out of position), and McClusky's decision to continue searching and subsequently stumbling across the Arashi, which Nimitz credited as "[deciding] the fate of our carrier task force and our forces at Midway".
In addition, it seems that Spruance, basically by coincidence, pulled out of his attempt at a counterattack before making contact with an IJN force that would have had a significant advantage against Spruance's force.
Presumably, very little of this was known prior to someone getting access to IJN records of their side of the battle - but at some point these details came out. Thus, my question is in two parts:
Is the impression I have of the battle being a fairly fortuitous victory that "shouldn't" have happened correct?
If the answer to 1) is "yes", how did US naval doctrine change in response to discovery of how much luck played a role in Midway?
So we can basically place Midway at sort of a halfway point for USN carrier aviation. It was at a point where real deficiencies had been identified through Spring of 1942, but where fixes were mostly temporary, or had not been fully disseminated. Midway would highlight several, and the down time after, and the carrier battles of the fall of 1942 would solidify them as points to be addressed. Then the force that was built up of new ships and experienced men beginning in summer of 1943 would see the fruits of their labors and the birth of the Fast Carrier Force.
In particular I want to touch on 3 points that all came into play at Midway and would see evolutions in doctrine. The number and usage of fighters in an air wing, the cohesiveness of an air wing as a single strike and distribution of deckloads, and the positioning of screens and escorts when multiple carriers were present.
Is probably the simplest issue, it was obvious to everyone even when the war began that the size of the VF or fighting squadron aboard each carrier was insufficient for the jobs required. There simply weren't enough Wildcats to provide a CAP that was strong enough to protect the ship and rotate, while also providing escorts to strikes. So from the 12-16 in December 1941, by June 1942 the average strength of a VF was 24-28. And even then that was barely sufficient really. It meant that maybe a flight of 4-6 Wildcats could be spared to escort outbound strikes, but then they would be forced to choose between protecting the dive bombers up high or torpedo bombers down low, all on a small fuel load. Coral Sea had also seemingly show the VB/VS dive bomber squadrons in need of more protection as they prepared to strike so by adding some confusion it only made matters worse. Ironically it would be the VF flight from Coral Sea veteran Yorktown that were the only ones to fly close escort of the VT squadron. Mostly because the fighters from Yorktown were a scratch group mostly drawn from the shore bound Saratoga air wing under LTCDR Thach. The F4F-3 that had been sued until then was also replaced for the first time at Midway by the folding wing F4F-4 that was finally arriving in large numbers. However the added weight of the hinge mechanism, and 2 more guns with less overall ammo(a move made mostly at the urging of the British) made it a rather unpopular ride and even more of a pig than earlier Wildcats and spawned quite a few sharply worded memos back and forth.
Fighter strength would still be an issue and mostly remain static at 30 available Wildcats per carrier on a good day for the carrier battles that Fall off Guadalcanal, in part because getting replacements was barely quick enough to absorb losses. But by the raid on Rabaul in late 1943 the Saratoga would sport 33 F6F Hellcats, and by June 1944 and the Philippine Sea the fighter strength would peek at the low to mid 40's per carrier with an attendant drop in the % of other types aboard.
Was mostly a technical question, mostly what was the best way to get a strike off, considering all the aircraft had different ranges speeds and ideal cruise settings, and that not all aircraft could fit on the deck at once to be launched. Often it could take the better part of an hour to get everyone on the road, would it be best to have a running rendezvous and aircraft form up enroute and risk missing each other, or orbit and waste fuel? Now as the ships and crews gained more practice the time required to conduct a full launch went down and new aircraft alleviated the range and fuel issues. But it was still a sore spot, as Mitscher and Hornet frankly had a just putrid showing when it came to handling a carrier at Midway that thankfully would not be repeated. By the end of the war the USN could easily coordinate great multi deck strikes forming up in the predawn dark and launch coordinate strikes with aircraft and effective range. But until then the running rendezvous would still be the most efficient method used in 1942 to maximize the useful fuel the aircraft had.
The final question of doctrine was in many ways the one that was fought the most. And that is when you have multiple carriers do you place them in one large formation with one combined screen or separate? Obviously Yorktown's being attacked twice aided TF-16 in avoiding danger, but the ragged nature of the strikes on Kido Butai show how poor coordination was by forces just out of sight of each other.
This would change for the landings in August on Guadalcanal. Fletcher had command of the carrier force, and operated 3 carriers in a single task force for the first time in the war for the US. It was in many ways a strong showing as dozens of individual flights and rotations protected the task force on the Southern side of the island, while also providing cover to 2 invasion fleets, and ground support to 2 invasion beaches on the North side of the island at Lunga Point and Tulagi. Fletcher however was not an aviator himself and many of the career aviators like George Murray, Jocko Clark, and John Towers took a dim view of his suggestions and memo's, while Fletcher knew it was not his field of expertise and often would allow the experts to do as they thought best if he was unsure. And eventually it was codified that the precious at the time irreplaceable carriers should be split up. This meant that at Eastern Solomons and Santa Cruise a single carrier would bear the brunt of the attacks and pay, while still the strikes were uncoordinated and failed to land true knockout blows. Once the carrier force was reborn in the second half of 1943 it would be finally accepted that multiple carriers could and should operate within a single screen, though ususally the fleet would be a series of 2-3 large carriers and a few light ones as opposed to one enormous one. Experience was also gained when HMS Victorious was loaned to the USN in 1943 and served with Saratoga for a few months.
Eventually Frank Jack's ideas would be codified into a series of memo's once he was recalled from the South Pacific, Halsey endorsed them after review, and so did Spruance. And in general they actually embodied many of the ideas above. Other factors like better communication between ship and shore based aircraft(especially spotting reports), more radar sets, new IFF gear to reduce CAP workload, and several others were also in play at Midway but would come to a head in the Solomons and take hard work and repeated efforts to improve.
Shattered Sword is a fine piece. But for what you are looking at OP I would suggest 2 works by John Lundstrom that cover the USN Naval Aviation community and operations following Midway and showing the reactions.
The First Team in the Guadalcanal Campaign is an operational narrative of the USN carriers in the second half of 1942. While Black Shoe Carrier Admiral follows Frank Jack Fletcher's tenure in command until after the battle of the Eastern Solomons.
We should also note that the USN didn't really get a good picture on how the other side had been fighting the battle until late in the war. The so-called "Nagumo Report" was not recovered and translated until Admiral Nagumo's death and the fall of Saipan in 1944. That would be the first real piece of operational intel the USN got on the Kido Butai in the battle. And even then it was incomplete and edited. The multiple interviews conducted with survivors after the war as part of the Strategic Bombing Survey would also contribute. But obviously by then the USN was the top of the game when it came to carrier operations, and many of the issues they faced in the battle were moot.
I should start off by saying that my source for everything I am about to write is Shattered Sword by Jonathan Parshall and Anthony Tully. If you are interested in the Battle of Midway it is the gold standard for books on the battle.
The notion that the American victory shouldn't have happened is overly simplistic. Shattered Sword compared overall aircraft strength between the Japanese and the USN, and the two forces were actually fairly evenly matched. The IJN had the advantage of having all of their planes in one concentrated carrier striking force, but the Americans had an unsinkable island air base on Midway which could absorb far more punishment than a fragile aircraft carrier.
Nimitz only committed his carriers to the fight after a careful consideration of risks and rewards. He didn't have to defend Midway. For logistical reasons the island would have been of little value to the Japanese, and it would have been a nightmare for them to keep a garrison there supplied. It's also far from certain that a Japanese landing would have succeeded. Midway was well defended by its Marine garrison and the Japanese had little experience or equipment for assaulting a defended beach. Look how much trouble they had seizing weakly defended Wake Island.
The problems that the Americans did have at Midway weren't the result of doctrine, they were largely from a lack of training and experience. Hornet's air group was almost totally ineffective during the battle. If both of Task Force 16's carriers had been able to launch a cohesive, coordinated attack rather than throwing planes at Nagumo's force piecemeal the battle wouldn't have even been close. The first blow from the USN would've shattered the Japanese fleet.