For example, during the kamikaze attacks of the Battle of Leyte against US escort carriers, were US aircraft engaging kamikaze pilots at low altitude while the Japanese pilots were in their final dives?
I am wondering because it seems like friendly fire would be a major concern for US forces if this were the case. I know that most AA defenses are normally in layers with interceptors engaging at longer range and surface-to-air fire being a last line of defense in most cases, but I couldn't find anything specifically about US doctrine for naval battles involving aircraft from this time period.
You are correct that AA defense was layered and the Kamikaze threat prompted an expansion of existing AA doctrine after Leyte. There were three rings of defenses for naval task forces. The outer ring consisted of a combat air patrol (CAP) and radar picket ships. The Navy swiftly modified destroyers and LCS to act as radar pickets. Each radar picket had its own fighter director that would work in conjunction with a CAP to vector fighters on incoming planes or formations before they reached the middle and inner layer. The middle layer of defenses were the 5-inch/38 guns of the fleet. Finally, the inner zone consisted of battery fire from the 40 and 20 mm guns.
The system worked reasonably well given that the Kamikaze attacks were not a wholesale departure from Japanese conventional attacks of 1943 and 44. The massive losses of experienced Japanese aircrew from the Solomons and other theaters meant that their replacements typically used small formations that flew roughly direct courses with minimal evasions. The Kamikazes often mirrored these earlier attack profiles. The Japanese did incorporate some of their hard-learned lessons from 1944 such as sending Kamikazes following US strike groups, attacking at high or low altitude, and using overland terrain to mask their approach. However, the flexibility of the US AA system coupled with the immense resources the US now had at its disposal limited the effectiveness of these new Japanese tactics. The sharp decline in Japanese pilot training also played a mitigating factor that limited the effectiveness of these new tactics.
The US's AA system was far from perfect and friendly-fire incidents could occur. Occasionally, the CAP could stray into the AA zones when pursuing Kamikazes. The picket ships' coordination with land-based aircraft and other friendlies outside of the carriers was often more ad-hoc and the CAP could be vectored in on these friendlies. Finally, the CAP sometimes wandered into the picket ships' AA (a mixture of 5-inch and 40/20 mm) as the Kamikazes began to go after the picket ships.
Sources
Friedman, Norman. Naval Anti-Aircraft Guns and Gunnery. Annapolis, Md: Naval Institute Press, 2013.
Rielly, Robin L. Kamikaze Attacks of World War II: A Complete History of Japanese Suicide Strikes on American Ships, by Aircraft and Other Means. Jefferson, N.C.: McFarland & Co, 2010.
Young, Edward M. American Aces against the Kamikaze. Botley, Oxford: Osprey Publ, 2012.
As you've mentioned, the usual plan was for defending planes to peel off as the fights closed in on the defending ships.
Having said that, there are numerous instances where fighters sticking to their targets despite friendly anti-aircraft fire. For example, the agony of the USS Laffey had at least one instance of a F4U Corsair following its target for so long that, after the destroyer shot the Japanese plane down, the Corsair accidentally struck the Laffey's radar mast.
So while the doctrine was to keep friendly planes away, it didn't always work that way.