The genesis of autocannons used in fighter aircraft started in the interwar period. There had been experiments during the First World War with semi-automatic cannon or modifications of ground-based AA guns, such as the Vickers 1 pndr Mark III and the COW 37 MM gun which only flying boats had the size and power to mount. Generally speaking, aviation designers used autocannon to target large, hard to kill targets like bombers, tanks, ships, or fortifications. Autocannons' slow muzzle velocity, rate of fire, and limited ammunition precluded them from being dogfighting weapons, although the idea that one or two hits could knock out an enemy fighter was a tempting factor in their development and use see this ground test video of a 30mm Mk 108 on a Spitfire fuselage for an example of how destructive 30 mm could be.

The adaptation of autocannon posed a number of problems for aviation designers. For one thing, they had significant weight issues. This was not only due to the gun itself, but also the associated weight from mountings, ammunition, and the various devices that minimized recoil. Especially in the 1930s, the limits of single engine power made these weight penalties quite troublesome. The simplest solution was to mount the autocannon in the centerline of the aircraft through the engine block. A mounting did not disturb the center of gravity of the aircraft and makes it easier to aim, thus offsetting the demerit of autocannons' limited ammunition. Centerline mounting had many tradeoffs though. Firstly, it only allowed for the mounting of one autocannon. Autocannons' ammunition feed systems were often persnickety and the rapid G-maneuvering of a fighter being thrown around could throw these systems out of whack. Second, the only inline-V engines could use this mounting system; radial engines could not use this form of mounting. This meant an entire class of fighter engine could no use centerline mounts. The third problem is related to the second in that the spacing between the inline engine cylinder blocks places a ceiling on the diameter of gun that could be mounted between them. This was particularly problematic for the Luftwaffe as the DB 600 series, the main inline engine, had a clearance of only about 70 mm. In addition to bore size, autocannons had associated systems like muzzle brakes, recoil springs, and cooling jackets. One solution for the Luftwaffe was simply have the barrel and muzzle extend beyond the propeller hub, such as the 20mm MG C/30L, but the barrel length posed problems. The result was that the Luftwaffe had to wait until the war to have a proper centerline weapon, the 20 mm MG 151, and even then it was relatively underpowered for its mission of bringing down four-engined bombers. The 30 mm MK 108 resolved that problem and could be fired through the engine blocks thanks to a novel design, but it created a host of new problems for the Luftwaffe.

One solution to the centerline problems was to mount the autocannon on the wings. This itself created another set of design compromises revolving around weight. The weight of the weapon plus its recoil put stress on the wing spar. The British Tempest, Firefly and Blackburn all had to be modified as a result of tests that showed their mountings could not absorb the recoil and their autocannon could damage the main spar. Housing ammunition also became problematic as the narrow chord of the wing could not house them completely, leading to a series of bumps that posed a drag penalty despite being aerodynamically streamlined. These problems also put a ceiling on the caliber of autocannon that could be mounted in the wing- usually around 20 mm. One solution was to mount the autocannon in underwing gondolas, such as the 40 mm Vickers S. The Germans also had a particular fondness for this expedient solution, but these field packs (Rüstsätze) often had significant aerodynamic penalties. Wing-mounted cannon, whether in gondolas or internal, also were less accurate because of wing flexing when firing.

Overall, to render autocannon viable weapons for fighters entailed a series of design compromises. A lot of this simply has to do with the weight penalties and their associated impact on the aerodynamic qualities of the aircraft itself. These weight penalties had less to do with the gun itself, although they were quite heavy, but rather all of the associated systems that allowed the gun to function as a automatic weapon. The Yak-9T is an extreme case in point because the mountings to absorb the recoil of the 37 mm NS-37 initially were so heavy they damaged the engine, so the cockpit had to be moved back. The Lockheed's solution to this problem in the P-38 was to amalgamate the gun, recoil mounting, and ammunition feed system into one single unit which eliminated complex separate housings for each.

As to whether or nor these autocannon were worth the myriad design compromises is a difficult question to answer. The temptation of "one shot, one kill" certainly led the Luftwaffe down a chimerical cul de sac of specialized heavy autocannon. The most extreme example is the "flying telephone pole", the 50 mm BK 5 for the Me-262 and Me-410. The temptation to for bigger caliber weapons only magnified their penalties. Jagdverband 44's experience with the BK 5 was hardly complementary, the weapon jammed frequently and its slow rate of fire and poor muzzle velocity was a bad fit with the Me-262's speed. Yet the BK 5 was only one example of a trend in the Luftwaffe's problematic approach to autocannon. The mid-war emphasis upon heavier caliber autocannon meant that Luftwaffe aircraft were simultaneously overarmed and underarmed. Even when these heavier autocannon worked as promised, their poor muzzle velocity and rate of fire meant that Luftwaffe pilots could seldom hit anything. The BF-109 K model incorporated the MK 108, but its poor muzzle velocity and unique belt-fed design meant it was difficult to aim and prone to jamming- leaving the pilot only with two 13 mm machine guns. The mixed caliber armament proved to be something of a problem in that heavy or light machine guns have different ballistic properties. The IJN's fighter doctrine stressed using the smaller machine guns first, and when they hit, switch over to the heavier autocannon. In practice, this distinction was hard to make in the heat of combat and many Japanese pilots could not manage this mixed system effectively. Mixed armament also posed logistical problems and pilots often had difficulty keeping track of how much firing time they had left. This complexity was one of the reasons why American aircraft typically did not employ autocannon above 20 mm. The P-39 and -63's 37 mm M4 was a notable exception and many US pilots disliked the M4 for the reason of the differential between the 50 caliber's ballistics. The most successful autocannon generally tended to be the more the ones that operated in conjunction with heavy machine guns that were of similiar ballistic performance in a balanced airframe design. This balance was easier said than done and the performance of autocannon armament in the Second World War was quite mixed- autocannon 30 mm and above had some impressive performance, but at the cost of significant operational penalties.

Source

Williams, Anthony G. Rapid Fire: The Development of Automatic Cannon, Heavy Machine Guns and Their Ammunition for Armies, Navies and Air Forces. Shrewsbury, UK: Airlife, 2000.

1. Though because of their great weight aircraft armed with cannon were not practical until the 1930s there have been attempts to arm aircraft this way almost since the beginning of military aviation. The first plane I know of that had a cannon in the nose firing through the propeller hub was the French SPAD 12 Ca.1 in WW1. The advantages of the system are that it only requires a single heavy cannon, does not significantly alter the centre of gravity in the plane (the engine already makes it nose heavy) and is very accurate and easy to aim. Compared to machine guns cannons cause more damage and have greater range. The former is relevant because as WW2 approached aircraft were increasingly made out of metal and innovations such as self-sealing fuel tanks made them harder to shoot down.

2. Given the longevity of the idea this is unlikely, cannon armed aircraft were emerging in every country during the time due to more resilient opponents and more powerful planes capable of mounting such heavy armament. Exported P-39's were certainly not the first nose cannon armed fighters the Soviet air force encountered, having been fighting ME-109s with similar armament for months before the entry of the USA into the war.

3. The increasing weight in armament was for the most part offset by the vast increases in engine power throughout the war. Late-war aircraft such as the Hawker Tempest were armed with four 20mm cannons and were still excellent dog-fighters. The Tempest of course was spawned from the Typhoon and was made into a fantastic high-altitude fighter by reducing the thickness of the wings not lessening it's heavy armament. As for bomber killers, the Luftwaffe relied on bomber destroyers almost exclusively armed with heavy 20mm and 30mm cannons to combat allied bombers. They experimented with larger calibres, even 50mm.

4. For single engine craft with higher calibre than 30mm cannons I can only think of the British Hurricane MkIID carrying a 40mm under each wing and the German Junkers 87-G carrying 2 37mm in the same configuration. Both were exclusively tank-busters.

German planes in general used higher calibers as their allied counter parts for the exact reason you stated, higher caliber was deemed more effective in fighting heavy bombers like the fortress. The main Luftwaffe cannon for this purpose was the MK 108 ( 30 mm ) going above this caliber the weapons were mostly used in ground attack aircraft like Ju 87 ( Stuka ). The 3,7 cm cannon was used in single engine aircraft for ground attack.

This higher caliber weapon made the German aircraft a bit weaker in dog fights with allied escort aircraft for example the P-51. The range of the Mk 108 was shorter which was compensated by its heavy impact when attacking bombers but a serious disadvantage in combat with agile enemy escorts.

The Luftwaffe fielded even higher caliber guns in aircrafts during WW2 like the 7.5 cm cannon but none of these weapons were used in single engine aircraft. The 3,7 cm cannon was the highest caliber used in single engine aircraft.

Excluded by your question is the Hs 129 with two engines which also saw a version equiped with 7.5 cm cannon in a single gun pod. This was heaviest gun used in a ground attack aircraft as far as i know.pic

Rifle caliber machine guns weren't terribly effective against bombers, so for a while there was a trend toward heavier guns. It was then realized that you had to hit the bombers to shoot them down, and those big guns had a low rate of fire and limited ammunition capacity, lowering hit probability. Better to hit a bomber with a burst of heavy MG fire or light cannon and hope for the best than to pop of a few shots from a 37mm with little chance of scoring a hit.

Here's what .303 machine gun fire does to an He 111.

Another.

Some info on German cannon.

Some info on German MGs.

hurricane with 2 40mm guns

4 - This question reminds me of a story I have been trying to dig up for the past hour. A rogue bomber pilot in I think the pacific theatre during WWII who essentially turned a bomber into a fighter, extra machine guys, and a heavy cannon included. Does this sound familiar to anyone?

edit: got rid of the massive print...