I understand 1942 was cancelled because it was obviously the height of WWII, but by 1946 the war was over. Only 2 years later the Olympics were hosted in London, one of the heavily bombed cities in the war.
I'm by no means a soccer historian, let alone a fan, but it looks like planning was key.
FIFA's first post-war Congress meeting was in Luxemburg in 1946. During the Rome FIFA Congress, a decision had been made to move the headquarters from Amsterdam to Zurich. There were meetings in France in 1938 and Germany in 1936. There were no meetings from 1938-1946. This also happened during World War I. PDF Warning FIFA Source
Also. the FIFA World Cup was still young. The 1950 tournament was only it's fourth, with the first three taking place in 1930, 1934 and 1938. In fact, the 1950 tournament is the first the British had participated in after years of refusing to be a part of FIFA. No one had even wanted to host the 1950 World Cup. Other info that can be cross-referenced with the FIFA link from earlier
At this point, I have no more facts. It looks like money and organization issues on top of being a fairly young event without a whole lot of enthusiasm. But some of this should explain why London was able to host the Olympics but had no interest in the World Cup - seeing as how they wanted nothing to do with FIFA until after the war.
Edit: Took out a piece of bad information that was contradicting.
Another edit: Looks like World War I caused a lot of issues for FIFA as well, which makes sense as FIFA in general was still young at this point, having only been created in 1904. Perhaps it sounds obvious, but it looks like World Wars have an adverse effect on international sports.