I’ve often heard that Operation Sea Lion was doomed from the start, mainly since the Royal Navy was just gonna blockade the channel. However, could the luftwaffe have been better utilized in fighting the Royal Navy to prevent this?
No, the Luftwaffe could not prevent the RN destroying any invasion of the UK. The Luftwaffe was poorly trained for attacking ships, would have too many competing duties to carry out, and had to contend with the RN's habit of ignoring dangers to complete a mission.
Attacking ships is a difficult thing to do at the best of times; they are narrow, moving targets, capable of making radical evasive manoeuvres. Warships also had their own AA defences, which, in the case of the RN in 1940, were heavy (though not as heavy as they would be later in the war). The majority of the Luftwaffe was poorly trained in anti-shipping operations. They had been trained to attack larger stationary targets - bridges and factories - or area targets like military units. This translated into a low accuracy against shipping targets. Dunkirk gives a good example of the weaknesses of the Luftwaffe in the anti-shipping role, especially against the destroyers that would make up the majority of the RN's response to an attempted invasion. During the course of Operation Dynamo, the RN deployed 40 destroyers. Of these four were sunk by air attack and a further six were heavily damaged by the same cause. In addition to this, the French Navy deployed nine destroyers, with one sunk by aircraft. The vast majority of these losses were from ships that were moored or were in confined waters (like Dunkirk Harbour), making them easy targets for the Luftwaffe. For a more general look, between the 1st April and 1st October 1940, thirteen British destroyers were sunk by German aircraft. Of these, nine were either operating in waters that confined their movement, or were moored. Over that period, a total of 85 German air attacks caused damage to British destroyers, a sinking rate of 15% - and this ignores many more attacks where ships were attacked, but were not damaged. In these 85 attacks, 13 destroyers were sunk, 23 were damaged sufficiently that they would be unable to fight, 13 received damage that would reduce their ability to fight, but would allow them to remain in the fight, and 36 suffered minimal damage and essentially remained in full fighting condition. In other words, just under 60% of destroyers attacked by German aircraft remained able to continue an engagement after taking damage. This low accuracy meant that the Luftwaffe could not prevent the RN operating in the English Channel should it need to.
Secondly, the Luftwaffe did not have the strength to carry out all the tasks it needed to. Its bombers needed to strike RAF bases, attack British ships in the Channel, attack British positions ashore, resupply any paratroopers and interdict British troop movements. Meanwhile, its fighters needed to suppress the RAF's own fighters, protect German shipping from British bombers, and escort German bombers carrying out the previously mentioned missions. Every one of these required a significant proportion of the Luftwaffe's strength. Carrying out all of them would have required spreading the Luftwaffe thin, greatly reducing its effectiveness. The Germans had difficulty suppressing British air operations over the Channel in any case. During the early stages of the Battle of Britain, the Luftwaffe attempted to destroy British convoys running through the Channel. While RAF losses were heavier than they would have liked, the Luftwaffe's losses were the same. Over the course of July and August 1940, the RAF suffered 150 losses, almost all suffered protecting British shipping and ports. In the same period, the Luftwaffe suffered 286 casualties. The Luftwaffe's best weapon against shipping, the Ju-87 'Stuka', was also its most vulnerable. It was a slow aircraft that had to fly straight and level during its attack run. During this part of the battle, and during its later stages, losses were high amongst Ju-87 squadrons, especially when unescorted.
Finally, air attacks on the RN were no guarantee that it would cease its operations. Looking at the Battle of Crete in 1941 is instructive. Here, the RN sortied to attack German shipping carrying troops to reinforce the paratroopers that had landed on Crete. As it did so, it came under heavy attack by the Luftwaffe. The Luftwaffe had many advantages here that it would not have had in the Channel in 1940. It had a specialist anti-shipping unit, Fliegerkorps X, plus support from Italian pilots well trained for the anti-shipping role. The regular units had also spent more time training for the anti-shipping role. The RN was operating far from its base in Alexandria, and so began to run low on AA ammunition after days of continuing strikes. Mediterranean Fleet was much weaker than the forces available for anti-invasion duties in the Channel, and its units were widely scattered, reducing the effect of its AA armament. British fighters were a non-factor, with the only ones available being 12 Fulmars from Formidable. The Luftwaffe had much fewer missions to carry out, and were thus more able to focus on the anti-shipping role. As a result, the RN suffered heavy losses during the fighting, with three cruisers and six destroyers being sunk by air attack, as well as damage to Formidable, two battleships, four cruisers and two destroyers. Despite these losses, the RN continued operations in the waters around Crete, destroying one invasion convoy, forcing another to turn back, and evacuating British troops once it became clear that the island could not be held. The RN's presence forced the Germans to abandon attempts to reach Crete by sea, at least until most of the island was in the hands of German paratroopers. The RN's performance is summed up well by two quotes from Admiral Andrew Cunningham, commanding Mediterranean Fleet. On the 24th May, he sent a signal to the fleet saying:
The Army is just holding its own…we must NOT let them down… STICK IT OUT
while a few days later, during the evacuation, he told his staff
You can build a new ship in three years but you can’t rebuild a reputation in under three hundred years,
justifying continuing the evacuation in the face of German air attacks. Given that this was how the RN behaved when attempting to defend a comparatively inconsequential Mediterranean island, it is not hard to conjecture how they would have behaved in defend of Britain itself.