How does any force that is out numbered win?

The Allies had thoroughly lost the mental battle for France long before Dunkirk and the Fall of France.

As soon as the German forces we able to through the Allies off balance and essentially operate inside their decision cycle allowing them to dictate the decisive points and the flow of the fighting. Through aggressive maneuvers and calculated risks of over extension by some key leaders they were able to mask weaknesses in their force and eventually through the entire Allied command team onto the wrong foot, and degrade morale, and consequently the fighting force as a whole. After the German army achieved a breakthrough at Sedan they Panzer units were able to utilize their faster speed to through the Allies off balance, and to essentially cut a swath through France and Belgium eventually to the coast.

That was a big problem because of the large number of units of the French, British, and Belgian armies now trapped in a giant pocket to the North of the German advance, that if not air tight, was becoming tighter and a wholly unexpected development. These were also some of the best units available, including most of the BEF, as the general scheme had been to allow a smaller number of troops to man the Maginot Line as a fixed strong bulwark and force the fighting to go instead towards Belgium and Northern France, however the Germans essentially prying the two wings apart was obviously now a big problem. And proposals for the trapped forces to fight their way out went nowhere and by the 3rd week of June Lord Gort the commander of the BEF had lost all faith in the possibility of a breakout, or for the combined force to set up and stay supplied in a large pocket to keep fighting and made his intention to withdraw towards the ports behind him for evacuation clear, with some French units joining and others fighting their own rearguard actions or trapped in pockets of their own while High Command in Paris was rotated and plans were created, changed, or abandoned, sometimes after attempted attacks or even limited success, with a growing cloud of almost defeatism in the face of sudden and powerful German advance backed by an aggressive air force and concentrated armor.

So the 400k men that were backed into the pocket around Dunkirk were really in no position to even hold their current ground let alone breakout. They were mentally beaten especially as it came to senior leadership having been pushed back and unable to stop the German breakthrough, had been bombed for weeks, and lost much of their heavy equipment in the retreats with command and control and supply threatening to breakdown along with communication between units outside the pocket to even coordinate any plans.

And I would also add some HEAVY caveats to the idea of an invasion of Britain. While if the Germans got a force the size of a Corps onto British soil, could hold a port and reinforce rapidly ti could be a major obstacle to kick them out. That is a ton of ifs and possibles. In the requirements of such an operation the Kriegsmarine was simple hilariously under equipped or prepared to undertake any serious attempt to land a force in Britain and then sustain it. They had less than 6 cruisers and barely 20 destroyers and 0 operational heavy warships in June of 1940 with which to gain control of the needed sea space. All this against what was still the largest navy on earth in the RN with 14 heavy warships to call on and several dozen cruisers and score of modern destroyers, even with air power factored in it is a tall order for any landing attempt to be in any way sustained.