There was no one process that was used to convert the Norse people but the first step always seems to have been converting the king or other important leaders.

It is impossible to know exactly what convinced them to convert since we only really have the written word of the Christian side of events. One of the most common elements is that people would often convert as a condition of peace following their defeat by Christian forces.

This was seen especially in England during the aftermath of the "Great Heathen Army" and its rampages across England. In the end the leaders of the army converted.

We also cannot dismiss the idea that the religion just simply struck a chord with these people. After all the idea of eternal salvation for everyone, and not just the warriors who fell in battle, would have had considerable appeal.

Hagiographies and histories of the Church are full of accounts of miracles being performed that won over their audience to the Christian cause, and while we cannot take them entirely at face value it is likely that missionaries had considerable success in winning over people.

There were also practical benefits to conversion, it opened up more trade opportunities with the rest of Europe. It also imbued the Norse kings with legitimacy in the eyes of the rest of Europe that they had not held as heathens.

Sources and further reading:

Dunn, Marilyn, The Christianization of the Anglo-Saxons c. 597-c. 700. This is not purely about the Norse, but it is a very similar context

Conversion and Identity in the Viking Age, ed. Ildar Garipzanov

and for a primary source,

Adam of Bremen, Gesta Hammaburgensis Ecclesiae Pontificum