Germany and Britain have similar population sizes as France but combined have fewer winners. Why does France have so many?
The Fields medal is one of the oldest truly international mathematical prizes. Many other notable prizes like the Veblen prize in Geometry (first awarded 1964) or the Wolf prize (created in 1978) are more recent, and more specific in their scope than the Fields medal. While there are some prizes awarded by national academies or mathematical organizations that will consider foreigners (such as the Bolyai prize or the Lobachevsky prize before the Soviet years), the Fields medal was international from the onset, since it is awarded by the International Mathematical Union. There was no serious competing award for the first 40 or so years of its existence. In addition, the early choices of recipients were quite judicious - people like Serre (FM 1954) and Selberg (FM 1950) went on to make many more contributions and remained influential long after their medal.
As for the reason for France's disproportionate representation, this is a frequent topic of debate. People (such as 2010 Fields medallist Villani) like to mention France's unusual education system, with classes préparatoires and grandes écoles. The curricula emphasizing math and physics (MP, MP*) enjoy particular prestige in France. The "standard route" for mathematically inclined students includes several notoriously difficult ranked examinations (entrance exams to École Normale Supérieure and Polytechnique, aggrégation), which emphasize abstract notions and rigorous study over a more applied and intuitive approach. According to some, like Villani, this environment is particularly fruitful to the development of mathematical research talent.
From a more historical perspective, one should note that French mathematics in general has had a huge influence on the subject since WWII. The École Normale Supérieure (where all French Fields medallists studied or worked at some point) has been a major center throughout the 20th century, and the private Institut des Hautes Études Scientifiques, founded in 1958, is considered the European equivalent to the IAS in Princeton, NJ.
In the 20th century, there was a general drive first towards more rigorous, and then more abstract mathematics, especially in algebra. The French style of abstract mathematics found its ultimate expression right after the war. This philosophy is summarized in the work of the Bourbaki group (the wiki article is decent, though opinionated, a few more pointers can be found here: 1, 2, or this nice Notices piece by mathematician A. Borel on his interaction with Bourbaki). The extraordinary successes of Grothendieck and Serre (both Fields medallists) in reformulating algebraic geometry in a very abstract form vindicated that abstract point of view, and that style of mathematics (abstract, definition-based, and conceptual) became very fashionable. The work of French (or French-educated) mathematicians like Serre, Grothendieck, Weil was not only important for its extraordinary content, but also because it guided mathematics as a whole towards more abstract fields like algebraic geometry and topology, whereas in the 19th and early 20th century it was dominated by analysis and problems from classical mathematical physics. The first Fields medallists, in 1936, Ahlfors and Douglas, were decidedly analysts, and so was Laurent Schwartz in 1950. Atle Selberg, the other 1950 medallist, can be considered a classical analytic number theorist. All this being said, I am not sure the taste for the abstract nonsense type of math is sufficient to explain French dominance. Apart from Serre, Grothendieck, and Lafforgue, the French Fields medallists worked in analytic fields, sometimes quite close to applications (dynamical systems, statistical physics, partial differential equations).
I have nothing useful to say about Britain, but when considering the relative importance of Germany in postwar mathematics (the medal was only awarded to two people before WWII), the Nazi purge has to be the focus of attention. Many mathematical minds who would be crucial for the later development of the subject (most notably Weyl) were forced to emigrate because of their Jewish heritage, usually to the States, or driven to suicide (Hausdorff). Goettingen, a major center of mathematics until the 1930s, was all but wiped out. Hilbert famously said "Mathematics in Göttingen? There is really none any more." when asked about the state of the department in the 1930s. Moreover, the attitudes of several prominent non-Jewish mathematicians with regards to Nazi influence and the drive to promote "German mathematics" ranged from ambivalence (Blaschke) to downright Nazi fanaticism (Bieberbach, Teichmueller).
On the other hand, several equally important Jewish mathematicians from France or who worked in France in the 1930s, and had to go in hiding or escape the country during the war, such as Schwartz, the probabilist Paul Lévy, Mandelbrot, etc... chose to return to their old positions and work in France after war, at least for some time.