There were many people in the United States who thought the space program was a waste of resources. You have to remember that much of it, especially Apollo, was happening during a period of great social unrest, the Vietnam War, and calls for increased Civil Rights. The space program was taking an immense amount of talent and money to (as the satirist Tom Lehrer joked) "put some clown on the moon," when there were problems right here "at home" that ought to have been addressed instead (like inner-city poverty, riots, etc.).

It's not clear that such a view was even "fringe": the Apollo program was never very popular. Again, this was not about a question of the dangerousness of the technology (or being "Luddites"), but was about spending priorities. Space research and travel is expensive. The sum total of resources available even to a very rich nation is finite. So there were many people, on both sides of the political spectrum, who said: "Why exactly are we spending so much of our money on this, rather than something else I value?"