I can't speak for average Americans, but in the case of Oppenheimer: amongst the educated elite of the 1920s-1930s there were many who were drawn to Eastern philosophies and religions for their apparent exoticism and difference. Oppenheimer's own dabblings in Hinduism and Sanskrit were very much of this sort, and if one were cynical at the time one could have seen it as pretentious (and some of his colleagues did see it in this way). Oppenheimer himself was a secular Jew whose parents were strong adherents to Felix Adler's philosophy of Ethical Culture which Oppenheimer found dull and tedious, so one might look also to that as a contrast. His Gita quote has been largely misunderstood.