Without a question, yes. Churchill spends a lot of time in his memoir detailing efforts to resist a landing, establishment of the Home Guard, etc.

The British had a very small army compared to the other continental powers, and a relatively large part of it was engaged in France as part of the BEF. The men were saved by the sea-lift at Dunkirk, but most of their modern equipment had to be abandoned (by this I mean the artillery, the surviving tanks, the supplies, etc.) Had there been no English channel, the German army would have rolled through Britain and Scotland without any serious problems.

The British were -rationally- concerned about invasion. They knew that it would be virtually suicide to cross the Channel in the face of the British fleet, since the Germans didn't have anything like the naval strength they'd need to contest the issue. But a British fleet that went into the Channel to stop an invasion would have been within easy striking distance of the Luftwaffe, and thus could have been destroyed, if not for the RAF being able to intervene. This is why one of the "conditions" for the German invasion plan, Operation Sea Lion, was achieving air superiority over southern Britain. If the Germans had managed this, then they could have attempted an invasion, and possibly done great damage to the British fleet; with a weakened fleet, a weak army, and a battered air force, the UK probably wouldn't have been able to hold off a second invasion attempt.

It's also worth noting that conventional military wisdom at the time was that forcing a landing on a defended, fortified beach was next to impossible. Churchill himself had attempted to do so in Turkey at Gallipoli in WW1 and suffered a bloody repulse. That the British were worried about landings simply highlights just how weak the British army was at this time - they didn't have nearly the strength to fully defend all the vulnerable beaches, or the guns to fortify them, in the same manner that the Germans did to the French coasts later on. The US and British developed the necessary weapons, doctrines, etc. to do it later, but those relied on absolute dominance in both sea and air, to a degree which the Germans simply never enjoyed.

At any rate, even significantly later in the war, when the Germans were engaged against the Soviets and the invasion risk was much smaller, the British generally retained significant forces in the UK just to make sure (and also because they only had so much sea-lift capability to get them elsewhere...) There were quite a few accusations that the British were content to have British divisions sitting in Britain while Australian, New Zealander, and Indian troops did a lot of the fighting in the Middle East, to the point that Churchill had to occasionally chivvy his commanders to reassign units so that more British troops were actually fighting.