They did consider such an invasion after the 1940 Battle of France and drew up Operation Sea Lion. During WWII it was vitally important to have aerial superiority for an amphibious landing. The first step for Germany therefore was to knock out the RAF so that the Luftwaffe could support the landings, and this was especially important since the German navy was completely outclassed by the Royal Navy. So the Luftwaffe would have had to negate the RAF so it would have a free hand to try to interdict the Royal Navy as well as support the troops during the landing.

The resulting clash between the RAF and Luftwaffe was the Battle of Britain. This was a battle of attrition, and the Luftwaffe lost. Sea Lion was postponed a number of times, then cancelled. With the RAF and Royal Navy intact there was no feasible way to cross the Channel with any kind of force that would have a chance of winning. After 1940 the concept lingered but was not seriously considered.

Certainly, even apart from the purely military aspects of RAF and RN opposition to the crossing, given the lack of proper landing craft and landing ships, a cross channel invasion could only have been carried out in a period of sustained calm weather. Once the summer was over and autumn was approaching the chances of that were increasingly remote so all the RAF really had to do was prolong the conflict until the weather was no longer suitable. By the following summer Britain would have been very well prepared to defeat any invasion, even if it was successfully managed, so the summer of 1940 was Germany's one and only chance of invasion. As shown below, if they'd tried in 1940 they would have lost.