The island configuration for carriers was present in carrier design from the 1920s. HMS Argus had a dummy island fitted to test the concept in 1918. Both HMS Hermes and, at least initially, the IJN's Hosho had islands. The island was beneficial for conning the ship, coordinating air-operations, and gunnery direction. Another benefit of the island configuration, recognized by the American and British first and later the Japanese, was that trunking up exhaust gasses through the island minimized its impact upon flight operations.

The early interwar recourse to flush-deck carrier designs was largely because of interrelated problems with weight and flight operations. The IJN removed Hosho's island after landing experiments showed that its masts presented a visual obstruction and wake turbulence that made it difficult to land. The island's real and perceived- remember this was when navies were still figuring out the science of air operation- negative impact on aircraft takeoff and landing meant there was some pushback from naval aviators and designers against islands. After their experience with both Lexington and Saratoga's island configuration, BuAer officials testified that the island's occupation of vital deck space and wind disturbances increased the chances of accidents and USS Ranger was initially to be a flush-deck design. Ranger did have an island, but one smaller and offset away from the flight deck. The island also posed a weight and size penalty and this became an increasing issue in the era of the Washington and London naval treaties. This was most evident in the design of Ryujo, which tried to cram as much of an airwing on as small a hull necessitated a flush deck. Not only would an island superstructure add more weight to an already overloaded design, but also hinder flight operations on narrow decks.

As it became increasingly clear that the naval arms restrictions were a dead letter in the mid-1930s, islands became a major component of carrier design for the three navies. As aviators became used to carrier operations, the island's fixed position was used as a point of reference during landing. The increasing importance of fire control for AA suites also underscored the need for an island to house these mountings.

The weight and size penalties of the islands were still an issue for escort carriers and conversions. The first British CVE's pioneered the use of small offset islands for navigation and coordinating flight operations. From the Attacker class onwards, island design became standardized. The IJN's CVEs and seaplane-carrier conversions kept the flush-decker design partly because of weight, but also because early experience with Hosho and other flush-deck carriers like Akagi and Kaga in their original configurations meant that IJN doctrine would stress directing carrier operations from a platform besides the flight deck. The IJN's novel system of landing lights for carrier landings also made landings on flush-deck carriers easier. Yet, the IJN's conventional carriers conformed to the international trend towards islands.

Sources

Friedman, Norman. US Aircraft Carriers: An Illustrated Design History. Annapolis, Md: Naval Inst. Pr, 1983.

Hobbs, David. British Aircraft Carriers: Design, Development and Service Histories. Annapolis, MD: Naval Institute Press, 2013.

Peattie, Mark R. Sunburst: The Rise of the Japanese Naval Air Power, 1909-1941. Annapolis, Md: Naval Institute Press, 2001.

So one thing that you will notice is that all the classes you mention are very much on the smaller side for carriers.

They were all either early converted or experimental designs, or were purpose built for escort carrier duty.

So that means they all had a much smaller beam to work with for having a flight deck ide enough to safely conduct flight ops. For instance USS Langley had a beam of just under 20 meters while the Lexington were over 32 meters! So on the smaller carriers there just wasnt enough space in many instances to have a large island and safely and efficiently operate as a carrier, hence some removing them completely, or having tiny islands.

You also then saw some intermediate sized carriers reconstructed to include them like Ranger and Kaga and Akagi. While the intermediate carriers on the smaller side had them removed, notably Hosho.

So in a sense the lack of a superstructure was never so much a design feature in and of itself, at least once ideas like multiple level flight decks were found to be impractical, and the IJN, USN, and RN began to mature as carrier operators. And that too big wasnt great either, the Lexington's were considered too large of superstructures as they obstructed pilot views and played havoc with the wind. But that an island as good for observation, fire control and command efficiency.

And more importantly all the navies realized they needed larger carriers regardless of Fleet/Escort usage because they needed more planes to be effective.

It also was very much a series of concurrent designs and trials, You have for instance Hosho, and HMS Eagle and Hermes all designed and built with islands from the start in commission at the same time as their peers by 1925, with the gargantuan Lexingtons following in 1927.