Where sports ever seen as a worldly sinful activity in the eyes of American clergy? Was it seen similarly to alcohol or homosexuality today (culture war)? How widespread was this view held?
There is evidence that New England Puritans banned secular entertainments (including sports) for moral/religious reasons. The General Court in Massachusetts outlawed shuffleboard in 1646, for example, and prohibited bowling in 1650. Boston banned folk football in 1657 (although that wasn't uncommon compared to Europe). There were also punishments for playing on the Sabbath. In general, Puritans seemed to discourage anything that took focus away from religion and education.
Allen Guttmann's "Sports: The First Five Millennia" is a good and accessible resource on this and other topics.