The main reason for these discrepancies in losses has much to do with the nature of the losses of these ships. The bulk of of the USN losses were from the Yorktown and Hammann. In the former case, the carrier sustained damage from two aerial attacks and it was a submarine hit that did her and Hammann in. The carrier was largely abandoned by I-168's attack aside from a salvage party and stragglers from her crew. This meant the damage the ship sustained was staggered, giving time for both evacuation and damage control.

Time was a luxury the Japanese carriers did not have at Midway once damaged. Three of the IJN carriers were disabled within a few minutes of each other, and the fourth, Hiryu, later that evening. While the losses in pilots and aircrew were surprisingly not that heavy for the three carriers, the rest of the carriers' compliment were not so lucky. The dive bombings started fires that raged throughout the carriers and the ammunition and avgas added further combustibles. The suddenness of the American attack also knocked out some of the carriers' officers and destroyed communications, making damage control a highly improvised affair. So there were sailors caught up amidst an ever-growing fire without any clear direction. the IJN tended to make DC the responsibility of specialists inside the engineering divisions. In the carriers, the IJN divided damage control between the engineering officer and the deck officer who were respectively responsible for DC below decks and flight decks. This meant only a very narrow clique of officers had knowledge or insight into how to contain damage once it had broken out. On top of this, DC training often interfered with their other duties, so that it tended to receive lower priority.

The design of IJN carriers also played a role in making this a dangerous situation. IJN hangars were closed and the second hangar deck was the strength deck of the ship. All large ships need a strength deck to protect them from the motions of the ocean. This was not much of a design problem for conventional ships, but carriers were a different matter. Naval architects usually have two choices for a carrier strength deck, the hangar or the flight deck. The Americans opted for a strength deck on a single hangar deck after experiments while the British preferred a flight deck arrangement. The Japanese instead had two hangars for their fleet CVs with the strength deck atop the lower hangar. This decision meant that Japanese hangars were rather narrow and enclosed (which was also a desirable trait for operating in Japanese home waters which often had inclement weather), unlike American CVs which were open. This design decision ensured that when bomb hits penetrated the unarmored flight deck and detonated inside this enclosed space, the explosions would rip through the ship and maximize damage. This was famously seen in Midway where after hits by dive bombers, Akagi, Soryu, and Kaga were immediately knocked out by bombs that exploded in the upper and lower hangar decks in the span of a few minutes. The magnified explosions damaged not only the hangars, but also a variety of subsystems like steampipes and generators. Yorktown's bomb damage might have been more severe had it possessed a closed hangar space, but she did not.

Aircraft carriers are highly vulnerable ships by their very design. They have to have fuel for planes plus ordnance for multiple strikes, and it is difficult to armor such a large area like a flight deck. Japanese carriers, both in terms of their design and personnel management, were more vulnerable to damage than their American counterparts. But American carriers were still vulnerable to suffering catastrophic damage such as the in the case of USS Franklin in 1945.

It's important to note that this problem wasn't limited to midway and carriers. The japanese persistently take higher casualties in ship on ship actions, even accounting for the total number of ships lost and the pronounced tendency of IJN sailors to go down with their ships. the deficiencies in damage control that u/kieslowskifan points to were an issue across the IJN. US ships were designed with elaborate protection against fires, with lots and lots of fire fighting equipment and a great deal of fire fighting training for the crew, the japanese ships were not and their crews were not, and the results tell. Read any account of battles and you read lots and lots of stories about IJN crewmen asphyxiating, nearly asphyxiating, or seeing their crewmen asphyxiate while fighting fires. Such accounts are far rarer on US ships.