Umberto Eco covers the topic thoroughly in The Book of Legendary Lands, 2013 (the English title sounds somewhat frivolous, but it is actually a work of history, as indicated by its Italian title, Storia delle terre e dei luoghi leggendari).

He traces the idea of the the hollow earth, which also seems to have also gone hand in hand with antisemitism, to Cyrus Reed Teed:

The idea was propounded by Cyrus Reed Teed (1899), who asserted that what we believe to be the sky (according to “the ignorant Copernicus’ gigantic and grotesque fallacy” and Anglo-Jewish pseudoscience) is a gaseous mass that fills the interior of the planet with zones of brilliant light. The sun, moon and stars are not heavenly bodies but visual effects caused by a variety of phenomena.

He shows that Peter Bender and Karl Neupert became proponents of the theory during the interbellum, and managed somehow to acheive access to the highest tiers of the military:

After the First World War, the hollow-earth theory (Hohlweltlehre) gained acceptance in Germany through the agency of Peter Bender and Karl Neupert, and was taken very seriously by high-ranking members of the German navy and air force, who were evidently sensitive to some extent to the occultist atmosphere that has been established among some representatives of the regime.

The specific incident that you are asking about seems to be a result of this work; one of the ways that the theory was applied:

Bender had suggested that the German navy make an expedition to the island of Rügen (in the Baltic) to try to identify British ships with powerful telescopes aimed upwards, along the presumed terrestrial concavity, using infrared rays.

The project was (naturally) an abject failure, and Bender ended up in a concentration camp for wasting the navy’s time and money. Nonetheless, the ideas didn’t go away, resurfacing in the German rocket program:

[S]ome V-1 rocket launches went wrong precisely because the trajectory had been calculated on the basis of a concave and not convex surface.