Early carrier designers understood the need for a superstructure on a carrier. This would provide space for navigation facilities, for controlling the air group, and for releasing smoke from the boilers. To this end, a number of island configurations were tried. The first carrier design, put together by Beardmore's shipyard in 1912, had two islands either side of the flight-deck. This maximised superstructure space, and simplified boiler ducting. By joining the two islands with a 'flying' bridge, the bridge could be located in the centre of the ship, improving visibility and making controlling the ship easier. This idea was tested, using scale models in wind tunnels, for Argus in 1916. These experiments, carried out in the National Physical Laboratory's wind tunnel at Teddington, suggested that this configuration would create severe turbulence. Real-world experience with Furious, which, in her original carrier configuration, had a large superstructure in the centre of the flight deck, showed that turbulence was a major problem for landing aircraft. This combination of small-scale and real-world testing ruled out the twin-island and central superstructure configuration.

Argus would complete without a superstructure. Instead, she was controlled from a retractable charthouse in the middle of the flightdeck forwards, or from 'wings' either side of the flight deck forwards when the charthouse could not be used. Smoke was carried away from the boilers by ducts which ran through the hangar and out of the ship aft. This proved to be unsatisfactory, turning the hangar into an oven and making approaches difficult. For future designs, the RN began to examine the use of single islands. This had previously been suggested by a Flight Lieutenant Hugh Williamson for Argus, but had not been adopted. Williamson's design had the superstructure on the starboard side. He appears to have been influenced in this choice by the flight characteristics of contemporary aircraft. These used rotary engines, in which the cylinders and crankcase rotated around a stationary crankshaft. The sheer amount of rotating mass introduced significant torque, making it easier to turn in the same direction as the engine was rotating. For most designs, this was to port, rather than to starboard. As such, pilots attempting to abort a landing would tend to turn this way. A starboard-side island would thus be safer. According to Brown, the British designers had this effect in mind. In 1918, during the redesign process for Furious, a single-island design was tested in the NPL wind tunnel. Originally, this had a port-side island, but this would be moved to the starboard side as the designs progressed. Ultimately, Furious would be rebuilt without an island, to maximise deck space. However, Argus would be given a mock-up island for experiments in October 1918. This was a wood-and-canvas structure, always positioned on the starboard side. It was found to be a considerable success, popular with both pilots and the ship's officers. The idea would be retained on all British carriers to follow.

During the 1930s, as the IJN was building up its carrier force, it constructed two carriers with port-side islands. Akagi would be rebuilt with one, while Hiryu would complete with one. This was an experiment inspired by studies which suggested that moving the island away from the exhaust gasses would reduce turbulence aft. Originally, they were to have islands on the starboard side forwards, but wind-tunnel tests showed that this would actually increase turbulence. A position amidships would be superior, but the funnels were found on the starboard side there. As such, the islands had to be moved to port. Japanese pilots found that this design did not improve the turbulence situation. Instead, it made things worse. Japanese designers acquiesced to their arguments, and the IJN returned to the starboard-side island. Akagi and Hiryu would not be rebuilt, though, as it was felt that the turbulence was not a significantly large problem to justify a lengthy rebuild.

Here's an aircraft-centric rationalization for the design decision. I know nothing about 1940s aircraft carriers, I'm familiar with more modern aircraft carrier operations.

The pilot of an aircraft traditionally sits in the left seat of a non-tandem aircraft. This led to a 'standard' aviation traffic pattern of left-hand turns. Making left hand turns means that the airfield (or aircraft carrier) remains on the left side of the aircraft while doing a landing pattern, and the left-seat pilot can keep constant contact with the runway, while the right-seat copilot's vision may be obscured by the cabin.

From the perspective of an aircraft landing on a carrier, given the pilot seating conventions and traffic pattern conventions, it is preferable to have the island on the starboard side. In order to land safely, practicality dictates that you need to see the runway location in order to fly to it, and see the entire landing area to ensure that the entire landing area is clear, has no obstacles and is safe. With an island on the Port side, the island would be between the aircraft and runway on a standard left hand pattern, and would obstruct the aircraft's view of the landing area. If the island is to Starboard, then the aircraft on a left-downwind can look at the entire flight deck to observe that it's safe to land.

From the perspective of the ship, controller in charge of airspace, and controller in charge of 'ground' flight deck operations, it's also preferable to have the island on the Starboard side. The 'Tower' controller needs to see both the condition of the ship's flight deck (to ensure everything is ready for an aircraft to take off or land), as well as the position of all the aircraft that are flying in the traffic pattern nearby. With the island on the Starboard side of the ship, the controller can look out the Port side of the island and see both the entire flight deck, and the entire standard traffic pattern. (Otherwise, that person would need to constantly walk from one side of the island to the other, and they could only see one piece of information at a time.)

Rotary wing aircraft are slightly different, but there have been enough rationalizations for them to keep the same island convention. Helicopters keep the pilot in command on the right side of an aircraft, which would make it more difficult to keep sight of a ship on a standard left traffic pattern. However, on landing, it's more important for a helicopter operator to look to the right side. While fixed wing aircraft look forward to land on arresting cables, helicopters land vertically on pads of the Port side of a ship. The key information on landing is the seeing the ship, and given the tradition of a rotary wing pilot on the right side of the aircraft, the Starboard island convention still gives the best visibility.

There are modern ships with control towers, if not 'islands' on the port side. Here's an amphibious ship with a smaller aft flight deck, and the 'tower' windows on the left hand side. The control tower is the three aft-facing windows slight angled down, and another window to the left, on the port side of the ship, just below the cannon. Note that this configuration still prioritizes simultaneously seeing the flight deck, and the port side traffic pattern.

If you're looking for sources on traffic patterns and operating conventions, search for US Navy 'NATOPS'. For example, here is an amphibious ship chapter. Page 4-31 gives a top down image of a helicopter carrier amphibious ship. This forum has an image from a full size carrier, and shows a Charlie pattern, the standard, left-hand pattern for a normal landing.

The following is more speculation on Kaga, but there's a huge advantage in having at least some asymmetric ships in a fleet. If you had two aircraft carriers, both optimized for left hand operations, you need to keep them rather separated in order to conduct simultaneous operations. If your downwind is half a mile away, your two aircraft carriers need to be at least a mile away from each other, which is far enough away that you need more escorts to protect them. However, if you have one aircraft carrier with a left pattern, and one aircraft carrier with a right pattern, they can be almost right next to each other and still conduct simultaneous take off and landing operations without interference, or complex real time coordination beyond de-confliction. With the ships being closer together, the escorts can do a better job. If there are any images of Kaga and other carriers conducting flight operations, or reports of relative positioning of Kaga and other carriers during flight operations, I'd overlay an estimate of their normal landing pattern (~1 mile forward, 1 mile aft, 1 mile either port/starboard depending on island position. Larger with high stall speeds, smaller with lower stall speeds), and see if there's an attempt to keep traffic patterns clear of each other.