I would say that to me the way it is used as an insult in the show seems wrong. Being able to write good poems, or kvad as the Vikings called them, was a good thing. Good skalds often got large and prestigious prices for making poems about kings and chieftains. As an example of this I will refer to Sagaen om Gunnlaug Ormstunge. It is an Icelandic saga taking place around year 1000, which is a little later than the time of that specific part of the show, but the general view would be more or less the same. In the saga Gunnlaug travels to Dublin where he meets the local ruler. Gunnlaug present to him a poem about the local lord and as a reward he is granted clothes and a golden ring as a reward. I will also note that when we are first introduced to Gunnlaug the saga is quickly saying that he is a good skald, which would imply that it is something that is important and well regarded, rather than an insult. Based on this I would say that skalds in general were well received, if they knew what they were doing they got a lot of gifts, and it would probably not have been an insult to be called a skald.

Now another question is if it was a good thing for a ruler to be a skald or if the art was to be left to other people. With my quite short search through parts of heimskringla(a historical work by Snorre Sturlason) I found at least one reference to the king Harald Hardråde making a poem. This combined with the fact that heimskringla in newer times is often seen as a kind of propaganda work for the Norwegian kings (there is reasons to believe that Snorre was biased in his retelling of the Norwegian history), would make it probable that a king making poems would not been a bad thing, since if it was a bad thing Snorre would probably not mention it.

Based on this I will conclude with my original statement: using the fact that Sigurd listen to music or plays an instrument as an insult is probably wrong. I say probably because there is still some problems here. Firstly, we don’t know if the skalds used instruments to perform their poems. In addition, we don’t know if the poems where sung or retold, so if they can be called music is also debatable, and therefore it is hard to conclude whether Ivar is in fact talking about skaldic poems, or some other kind of music, and in that case the insult could still be valid. Also, if all Sigurd do is listen to or perform music the insult could be the fact that he is not raiding or fighting rather than the fact that he is listening to music. This would probably not be well received by the men, who would probably not want to follow Sigurd if they agreed with the statement.

Thank you for reading and I hope my English was not too bad. It is not my first language and I hope you can excuse the mistakes I have done through the answer. If something is unclear please ask, and I will try to explain it better.