I decided to read The Scarlet Letter. The prologue is an autobiography where the author states that his ancestors were Puritans in Salem. He imagines his ancestors would ask about himself, "What is he? A writer of story books! What kind of business in life - what mode of glorifying God, or being serviceable to mankind in his day and generation - may that be? Why, the degenerate fellow might as well have been a fiddler!"
Would have been being a fiddler during the time of the Salem witch trials really have been considered a degenerate profession?
For the most part, yes, although I don't think that "look down on" is the most accurate description of the sentiment. Puritans saw music and other art forms as a distraction from the pragmatic necessities of living a godly and productive life, rather than as some kind of overt social or moral failing.
The earliest Puritans — in keeping with their Calvinist origins — rejected the Catholic conception of ritualized worship that appealed to the senses, including liturgical music, and generally extended that to their lives outside of church, as well. Later in the 16th century, but still well before the Pilgrims (1620) and the Great Migration (1630-1640), the Puritans began to adopt the new English-language psalm books that began to circulate during the reign of Elizabeth I. This addition was justified as worship rather than art, but still caused a bit of doctrinal conflict, as such things often do.
There were fully-stocked music stores operating in Boston by the early 18th century, so the strictures seem to have loosened, at least pertaining to activities outside of church.