I've heard that it had a lot to do with the cliffs of Britain but it seems with about 4 years between the conquering of France and D-day taking place they would have had sufficient time to come up some sort of plan.
There are several reasons why the Germans were unable to invade Britain during WW2. None of them were the White Cliffs of Dover, or that they didn't have a plan. At its most basic, the reason the Germans didn't invade Britain isn't that they didn't have a plan, it's that they were fundamentally unable to carry it out.
The German plan was called Operation Sealion. Planning for the operation began following the fall of France in June 1940, primarily within the German Army's high command. A first plan, drawn up in July 1940, called for an assault along almost the entire length of the British south coast, from Ramsgate in the east to Lyme Bay in the west. This plan called for an initial landing of some 90,000 men and 650 tanks, followed up by a landing of a second echelon of 160,000 men. This would be merely the first, but largest, of four waves. Ultimately, this had to be drawn in to a narrower front, as the Kriegsmarine (the German navy) didn't feel it could protect and supply an attack on such a wide front. At the end of August, a new, less ambitious plan was drawn up. This called for a much more limited landing in Kent and Sussex, with a total of 9 divisions to be landed. However, the operation would ultimately be cancelled.
The biggest reason that Sealion wasn't carried out is the Royal Navy. This hugely outnumbered the German Kriegsmarine. As a simple example, Nore Command, one of the RN's four home commands, and the one where the invasion would have fallen, had 32 destroyers in August 1940. The Kriegsmarine as a whole had just ten. The RN had far more cruisers than the Kriegsmarine, and was well prepared to commit battleships - Revenge was moved to Plymouth in preparation for any invasion, while Hood and Nelson were retained on the Humber for much of 1940. The Kriegsmarine had to carefully husband its limited supply of capital ships - it had just three in 1940, two of which were outclassed by their British counterparts, and one of which was working up and in no condition to fight. The Kriegsmarine could have committed submarines to stop the RN. However, the shallow, confined waters of the English Channel aren't the best for submarine warfare. Additionally, to stop the RN, it would have required such a significant portion of the Kriegsmarine's U-boat force that the Atlantic convoy routes would have been practically un-threatened. This would have both allowed the RN to commit the convoy escorts to the battle in the Channel, and allowed the British to receive much-needed shipments of food and arms. There was no effective way for the Kriegsmarine to win even temporary, local, naval superiority, which would have made a landing possible.
The second big reason for the cancellation of Sealion was the German failure to win air superiority over Southern England. Air superiority was vital for such an operation. With it, paratroopers could be landed to seize strategic points, attack aircraft could strike at communications links and troop concentrations, and dive bombers could threaten any ships moving in daylight. Air superiority was particularly essential for the Germans - it could somewhat counteract the weakness of the Kriegsmarine, and could lessen the need for heavy equipment to be shipped across the Channel. However, the Luftwaffe was unable to win the Battle of Britain. The RAF's air defence network proved to outmatch the German capability to destroy it. Even if this had been achieved, the RAF could simply withdraw to its bases in the North, which the Germans couldn't strike without taking heavy losses, as their fighter escorts didn't have the range. From these unscathed bases, the RAF could flood southwards when needed, disrupting German air superiority. The German inability to win and keep air superiority made an invasion of Britain near impossible.
Finally, the Germans didn't have the experience or equipment necessary to carry out such a landing. Large amphibious invasions require a lot of staff work to determine the best landing beaches, the shipping requirements, and other such details. The Germans did very little of this staff work. The Army assumed that the Channel crossing would be much like a river crossing, and treated it as such. This meant that they underestimated the logistical requirements, and overestimated the Kriegsmarine's ability to fulfil them. They also ignored the terrain behind the beaches- one of the landing beaches, around Romney in Kent, had a marsh just inland of it, backed by a strong defensive position in the Royal Military Canal. Meanwhile, the Kriegsmarine's staff were struggling to find the shipping and harbour space they would need. The estimates were that the first wave would take roughly a million tons of shipping. A third of this was available in France, Holland and Belgium. To find the rest, about a third of Germany's merchants, all its trawlers and all its tugs had to be requisitioned, along with many of their Rhine barges. This meant that the Germans had to reduce shipments of food, coal and iron ore, and delay their shipments around the country. There were also problems with harbour space, as the ports along the French, Belgian and Dutch coasts could only just hold the necessary shipping. This problem was exacerbated by British bombing and naval bombardments.
The Germans were underprepared and woefully underequipped for any amphibious operations in Northern Europe during WW2. The best way of showing this is to compare Sealion with its Allied counterpart, Operation Overlord. For Overlord, the plan took two years - for Sealion, it was just three months. For Overlord, the Allies carried out a thorough investigation of landing sites, using special forces beach reconnaissance, aerial photos, French resistance contacts and pre-war sources. The Germans did nothing of the sort. The Allies brought amphibious experience from North Africa, Sicily, Dieppe and the Pacific. The only comparable German operation was the shambles of Operation Weserubung, the invasion of Norway. The Allies had both air and naval supremacy, something the Germans could only hope to achieve. The Allies had a vast fleet of ships, including 6 battleships and 25 cruisers, for shore bombardment, the Germans had a few destroyers. The Allies had over 4000 specialised landing craft, the Germans had a few hundred converted barges. Compared to Overlord, Sealion looks like what it was: an impossible pipe-dream.
Sources:
The Defence of the United Kingdom, Basil Collier, HMSO, 1957
Operation Sealion: An account of the German preparations and the British counter-measures, Peter Fleming, Pan Books, 2003