I'll try to explain, but I tend to be rambling when I write, but hopefully it'll make some sense!

While we might think of the vikings as ruthless individuals, having to rely on their family for protection or justice, living in a lawless world where there was no centralized judicial systems for penalizing individuals or protecting them, the reality of it was that the vikings relied heavily on laws. Laws were important, and breaking them was not looked upon lightly.

Frostatingsloven (originally the law of the Frosta ting, covering the Trøndelag disctrict in Norway, established at around 400 BCE and put into writing and amended in 1274 by King Magnus Lagabøter) famously put it like this: With law the land is built, and no one shall break the law, but those who will not offer others the law shall not be granted law themselves."

Ting was the name the vikings used for their courts and judicial gatherings. Criminals were convicted at the ting. Allegations would be put to someone by a group of people (known as kvidr by the vikings), putting forward and describing the facts of the case.

A jury of twelve free men, in more important cases it could be 2 or three times twelve, determinded the question of guilt. A law sayer, lovsigemannen told the jury what the law said about the crime the person was accused of. Then the jury either convicted or freed the accused. If you were convicted you could, in most cases, pay a fine and the matter was over with, if you couldn't pay or refused to pay, you would be declared outlaw / fredlaus. Especially if someone had stolen from the body, used weapons that were defamatory and humiliating (like stones, sticks of wood etc), or somehow killed in a way not appropriate, they'd run a high risk of being declared outlaws. If you did kill someone you had to inform the community of the murder as soon as possible, killing someone in secrecy or trying to keep it a secret was very much a bad thing.

Again, in the words of Frostatingsloven, "those who will not offer others the law shall not be granted law themselves." This is a key to understanding the fredlaus / outlaw system. Fredlaus literally means without peace. Someone had broken the law, and thus put themselves outside of the law and society. You had shown through your actions that you didn't respect the law, now the laws didn't apply to you anymore! You were outside of society, outside of the law, you lost your property, anyone could kill you without repercussions. Someone helping or supporting you would also be dealt with harshly. There were no centralized powers or structures that could reliably enforce law and order in any meaningful, so it was left to the communities.

The basic principle of the laws was that the punishment should be equal to the action that was being punished. If someone is killed, a balance is achieved when the killer himself is killed. The old norse laws describe killing as a revenge for killing not just as fair, but as a positive thing.

For example, Gulatingsloven mentions that you could not accept monetary settlements for murder more than three times in a row, unless you had revenged the killing(s) in the meantime. As one can imagine, this led to rather tragic chains of honour killings, and during the 1200's the laws changed, and the kings starting the demanding the matters were settled with fines, and not another murder.