There are few naked-eye visible galaxies. Other than the Milky Way, the only galaxies brighter than 6th magnitude are the Large Magellanic Cloud and the Small Magellenic Cloud, the Andromeda Galaxy, and the Triangulum Galaxy. Of these, the LMC and SMC, both nearby satellite galaxies of our own, are fuzzy patches of light, and Andromeda is a fuzzy elliptical blob. These three were known long ago (but the LMC and SMC are southern sky objects, and not known to ancient peoples in the northern hemisphere). None of the three has a naked-eye appearance appearance that would suggest a swastika.

Triangulum, seen through a good telescope, has a much more visible spiral structure. However, it is much dimmer than Andromeda, towards the limits of what can be seen with the naked eye, and its spiral structure is not apparent to the naked eye.

The spiral nature of spiral galaxies was only recognised in 1845 when William Parsons observed the Whirlpool Galaxy (M51) using a 72" telescope. Earlier telescopic obsevations (e.g., when Messier discovered it in 1773 - it's too dim to be seen with the naked eye) failed to see the spiral structure. Without seeing the spiral appearance of spiral galaxies, there would be no reasons to symbolise them with a swastika.