Oh man, if you'd asked this question last week I'd have made the students from my Viking World class answer it for extra credit seeing as how we spent a not inconsiderable amount of time on this particular topic. Unfortunately (or fortunately depending on who you ask) the quarter just ended so I suppose I'll have to take a crack at it myself.
So... you've stumbled on what we would term not just a historical question but a major historiographical one as well. The question of Viking violences is a key historical debate amongst Early Medievalists and has been for over 60 years. I'll try to trace out some of the key actors/moments in the debate and use that as a way of addressing the historical question as well.
Buckle in, I'll try to boil down a two-hour + class into a coherent reddit post!
So first off: why do we think Vikings are violent? Well, the obvious answer is that the very term "Viking" means pirate or raider and we have ample evidence that this activity was a key part of Scandinavian engagement with the wider world. Our first "Viking Age" (late 8th- mid 11th century) written sources about "Northmen" certainly seem to give us that impression. The most famous is probably the raid on the monastery of Lindisfarne in 793 which is immortalized in a series of letters by Alcuin (and recreated in the second episode of the tv show Vikings incidentally) but the actual first record of an attack comes from slightly earlier in the Anglo-Saxon Chronicle entry for 789 (or maybe 787...) which notes
This year, King Beorhtric [of Wessex] married Eadburg, Offa's daughter. And in his days there came for the first time three ships of Northmen, from Hordaland [in Norway]. Then the Reeve rode to meet them; he intended to have them go to the king's town because he did not know what they were. They killed him. These were the first Danish ships to attack the land of the English people . . .. (Source: trans. A.A. Somerville, from David Dumville and Simon Keynes, general eds., The Anglo Saxon Chronicle: A Collaborative Edition (Cambridge: D.S. Brewer, 1983- ), vol. 7; MS E, ed. Susan Irvine (Cambridge: D.S. Brewer, 2004), pp. 41 - 46.)
Around this same time at the close of the 8th century the various historical sources (the aforementioned Anglo-Saxon Chronicles, the Annals of Ulster and the Royal Frankish Annals being the most notable) all record a number of attacks on monasteries and issues of piracy and this is reflected as well in various legal codes, letter, and other docs which reference piracy as well.
Throughout the ninth century, but most notably from the mid 9th century our various Frankish (Continental), Irish and English sources record numerous raids and attacks which escalate from quick lightning raids to multi-seasonal ventures to full scale sieges and invasions. On the Continent alone we get somewhere in the range of 120 individually attested attacks between 840 and 911 and that's not even to mention the "Great Heathen Army" that conquered chunks of England or Viking activity in the British Isles, the Baltic etc.
So there is lots of raiding going on but beyond just the sheer number of attacks the image we get of the nature of those attacks also paints a picture of a truly terrifying and blood thirsty people. Alcuin writes in his letter to Abbot Higbald of Lindisfarne:
Now that I am away your tragic sufferings daily bring me sorrow, since the pagans have desecrated God's sanctuary, shed the blood of saints around the altar, laid waste the house of our hope and trampled the bodies of the saints like dung in the street.
The various annals I mentioned tend to be pretty dry but they consistently refer to towns being burned and devastated and people being carried off or killed (sometimes both). And sometimes they get into high color detail, as when the Chronicle of Nantes notes that the Northmen:
captured the city of Nantes and destroyed the holy cathedral of God. In the midst of their violence they even beheaded the Bishop Gunhard, who stood at the very altar praying the “Sursum Corda” [the opening prayer of the Communion] and then slaughtered all the people gathered therein for mass.
Other sources, such as vitae (also known as hagiography, biographies of Saints) and miraculae (records of the miracles of already dead saints) likewise give us a pretty vivid picture of Viking violence. Walafrid Strabo, a 9th century Carolingian intellectual tells us in his Martyrdom of Saint Blamthac that after the Saint refused to hand over his church's relics the Vikings tore him "limb from limb." In the famous Old English poem The Battle of Maldon the Northmen are referred to as "wolvish Vikings, avid for slaughter" (trans: Kevin Cross, The Battle of Maldon and Other Old English Poems).
So our non-Scandinavian sources paint a pretty vivid picture of Vikings not just as raiders but as particularly cruel ones at that. This image of violence is also present in later (Scandinavia is largely pre-literate until the 13th century) writing as well. Skaldic poetry is filled with vivid images of violence and slaughter. A poem attributed to Arnor Thordarson from the 11th century about the Earl of Orkney's campaign into England claims:
The earl bore his banner / on to ancient English ground, / his troop reddened eagle’s tongue / as he made flags advance. / Flames shot up; halls collapsed; / his men made refugees; / the foe of boughs [fire] spewed smoke / and threw light towards the sky. (trans. Judith Jesch in Viking Poetry of Love and War)
The Old Norse sagas are filled with violence and feuds and of course who hasn't heard of the Blood Eagle, the ritual where Vikings apparently ripped out your lungs attached to your spine to make a beautiful semi-abstract eagle of blood on the snow?