It's to my knowledge that 'nothing' actually happened to them. The foremost trait of Viking culture was raiding. It's my understanding that Vikings ceased when the raiding ceased. What happened in Northern Europe during the middle of the 11th century to halt Viking raids, and essentially putting an end to Vikings as a whole?
"Viking" was originally a word that meant simply "pirate"—although both scholars and the public now use the word to describe a "Viking culture" that spread with the Norse diaspora from the North Sea to a broad swath of the North Atlantic and Eastern Europe. But we know that North Sea piracy existed both before and after the "Viking Age", so in some ways, this is more a question about perspectives on violence than it is about viking activity itself.
Around the year 800, Norse pirates began to take on increasing importance for writers recording the events of their day. These writers actually referred to these pirates most often as "Danes", by which they seem to have meant anyone who spoke the Old Norse language or could otherwise be associated with speakers of Old Norse. It certainly didn't have much to do with the "Danish" kingdom, since there were certainly many kings who spoke Old Norse, although Frankish sources do tend to show most interest in the petty kings based around modern Copenhagen.
The problem seems to have been that by 800, Francia, Anglo-Saxon England, and to a lesser extent Central Europe and the rest of the British Isles, were increasingly falling under the sway of land-based aristocracies. These were elites who gambled that the large-scale organization of agriculture could produce enough of a surplus for them to exploit and to set them apart from workaday farmers. This was a gradual turnaround from the economic depression of the early 600s, and it took hold at different places in different times, but by the late 700s, the Continent and the British Isles were both populated by aristocracies who saw their power as rooted in an ability to control the landscape.
Enter the problem of pirates. Pirates do not claim a land-based source for their power, although their activities certainly can challenge land-based powers. Even so, they don't seem to have bothered the West European aristocracies all to much even through the early 800s. Monks writing histories howled about viking raids, since they were frequent victims, but kings paid little attention. The Anglo-Saxon kings were more interested in fighting amongst themselves, as were the descendants of Charlemagne on the Continent.
But by about the 860s, things really turned around. Piracy had proven to be very profitable, and veteran pirates were beginning to take on the pretensions of being petty kings. They found the opportunity to establish new kingdoms based on maritime power at places such as Dublin, York, and Normandy. Meanwhile, the old landed aristocracies saw the very basis of their power coming under threat. After all, if a landlord can't protect his property from seaborne raiders, what kind of a landlord is he?
That gives us two competing systems of power, but systems that were also gradually converging into a new shared direction. Ultimately, the old aristocracies largely won out, albeit much changed. Alfred the Great famously established a fleet in England, and the Carolingian Empire permanently broke up into a number of smaller, more manageable kingdoms. Meanwhile, the maritime societies that survived were likewise changed. The Danish kings, and subsequently other Norse communities, adopted the organization and ethics of land-based powers. This notably included the spread of the Christian church, whose institutions were at that time inextricable from the extension of landholding. With the expansion of the Church came a stabilization of literary culture; monks no longer saw themselves as falling under acute attack, and the shrill tenor of the early Viking Age gradually disappears from their chronicles.
It's useful to contrast the traditional beginning and end of the Viking Age. In 793, a band of just a few ships did a hit-and-run raid on the English monastery at Lindisfarne. In 1066, a major army led by an established Norwegian king invaded England but was put to flight at Stamford Bridge; the Anglo-Saxon force that survived then marched south where they were defeated by William of Normandy, a descendant of yet other viking raiders who now headed an army intent on expanding their landed possessions. Between these events, we see the difference between small-scale independent pirates and large-scale state-organized violence rooted in the exploitation of surplus agricultural production.
So this is in some ways a story about the differences between Western European societies and North Sea maritime communities gradually merging into the hybrid societies of the Central Middle Ages. Piracy had had its day, and its leaders became more like land-based aristocrats, even as land-based aristocrats became more interested in how they tied into maritime communities. The next major maritime community to develop in the North Sea area, the Hanseatic League, prospered because it found opportunities to collaborate rather than compete with kings. Piracy still existed, but the world where viking pirates threatened to unravel the whole fabric of northern Europe had come to and end.
This is, of course, a huge subject and is discussed in many places. My interpretation of events is by no means the only one, although I believe it represents the growing consensus of the field. I find the easiest entry points and summaries of current debates to be the two articles below:
James Barrett, “Rounding Up the Usual Suspects: Causation and the Viking Age Diaspora,” in Atholl Anderson, James H. Barrett, and Katherine V. Boyle, eds., The Global Origins and Development of Seafaring (Cambridge: McDonald Institute for Archaeological Research, 2010), 289-302.
James H. Barret, “Maritime Societies and the Transformation of the Viking Age and Medieval World,” in James H. Barrett and Sarah Jane Gibbon, eds., Maritime Societies of the Viking and Medieval World, Society for Medieval Archaeology Monograph 37 (Leeds: Maney Publishing, 2015), 1–13.