For this answer I will mainly be using the idea of Anarcho-Capitalism as explained by David Friedman. This is because his essay "Private Creation and Enforcement of Law: A Historical Case" is pretty much the seminal work in the analysis of Icelandic law as applicable to An-Cap. So what is this definition? It is a political system where there is no government and all of the (currently) government controlled entities such as police forces, courts, schools, etc. are run as private corporations in a free market environment.
The institutional framework of anarcho-capitalism [is where] private, for-profit agencies offer sets of legal codes and enforcement of [those] codes. In anarchy no institution exists with legitimate authority over all members of society. (Daniel Sutter)
So how should we define government? If anarchy is defined as a lack of government, we need to know what sort of system we are comparing against. David Friedman, in the monograph "The Machinery of Freedom", defines government as:
An agency of legitimized coercion. I define 'coercion', for the purposes of this definition, as the violation of what people in a particular society believe to be the rights of individuals with respect to other individuals.
Legitimized coercion is a situation where the rights of an individual can be violated without repercussions to the violator because they are part of the system. A cop holding me at gun point and ordering me around is (to a certain extent) condoned and potentially even encouraged. If I pull a gun on a cop though, I am a criminal and will be punished. This unbalance is legitimized coercion.
For Friedman the ideal An-Cap world would involve an assortment of protection agencies and courts, each providing different services and sets of laws. Individuals would be able to pick and choose which agencies were the best fit for them and if they were unsatisfied they could end their contract and move to a different provider. If two people were contracted to agencies that utilized different courts (and thus different sets of laws), then the agencies would reach some sort of deal on which court to use or seek arbitration from a neutral party.
So now that we have given quick definitions of government and an anarcho-capitalist ideal, let's look a bit at Iceland.
Iceland was settled in the late 9^th century. The island (as far as we know) was uninhabited by humans before the first wave of Norwegian settlers. This resulted in a situation where the settlers could choose pretty much anywhere on the island to set up a homestead and claim as much land as they thought they could handle. In a sense this initial period appears to have been true anarchy. That is not to say that it was chaotic but rather for the first ~60 years of the settlement there was, more or less, no system of law or government at all. Finally in the year 930 a group of men founded the Alþing. The Alþing was a gathering of constituencies from around the Island where laws were determined and various court cases would be taken care of. So what were these constituencies composed of?
The head of the constituency was the goði. This was a position that was initially most like a priest. He had vague authority over a group of people, called a goðorð. As time went on the role of goði also came to be more applicable to the modern idea of a chieftain, though still not completely analogous. When a goði went to the Alþing he would bring along a contingent of men from his goðorð while the rest of the men would be expected to pay some amount to cover the travel costs of the men going to the Alþing.
As a goði a man was entitled to a seat in the lögretta which was the group that determined the laws. In addition, if there was a court session that needed to be held the goðar were able to appoint their þingmen as judges (there were very specific guidelines on how judges were able to be assigned based on the particular court that the case was being held at).
Now in a society such as this, with a bunch of chieftains running around, you would expect some amount of territoriality. This appears, for the most part, to have not been the case. While the law required farmers to be associated with a goði, there was nothing that said they must be associated with the closest goði. They could choose any goði within the quarter that they resided in (Iceland at the time was divided into four quarters based on the cardinal directions). If something occurred where the farmer wound up unsatisfied with the goði that they had chosen, they were free to switch their association to a different one and did not have to move.^1
The really interesting thing though is that the position of goðorð was basically a marketable good. They could be inherited, bought, sold, etc. Thus, not only could a þingman decide that he wanted to be associated with another goði, but if he had the goods he could purchase a goðorð for himself and all the benefits that entailed.
In terms of the court system itself there was nothing resembling the groups of courts, each with its own set of laws, that Friedman envisions as the ideal An-Cap situation. Each time a case, or series of cases, is to be heard each goðar picks a number of þingmen to make up the judges. So theoretically each case could be heard by a completely different set of judges, each with their own ideas on how the law should be applied. In addition, there were no "protection agencies" or privatized police forces. Enforcing the will of the court came down to the individuals involved and whatever back up that they could gather together from their family, friends, and connections within the goðorð. And while this may seem prohibitive for people with less connections or less money/goods, the cases could actually be sold. So if a poor farmer could not afford to prosecute for whatever reason he could sell his case to the highest bidder.
Another notable feature is that there was a second group that most farmers belonged to. This was the hreppr. This was a collective of at least 20 farms that functioned as a safety net. They would provide care to the poor, insurance in case of accidents (such as a house catching fire), and other services along those lines. Hreppr were based on where you lived and once you had joined one it appears that you had to stick with it unless you moved.
So while it does not fit the exact mold of the ideal An-Cap society (as imagined by Friedman) it definitely had some similarities. No central government (lack of "legitimized coercion"), marketable positions within the law making body, privatized welfare, the ability to choose which authority figure (goði) to associate yourself with. And it lasted for quite a long time. 930 until 1262 or 63.
But was it actually functional? When we look to the sagas we see that:
Although medieval Iceland had a complex court structure, most disputes did not lead to adjudicated outcomes…[Heusler] counted some 520 cases, of which 297 led to vengeance, 104 to arbitration without prior recourse to law, and 119 to lawsuits. Of the 119 legal actions only 50 led to a judicial decision; 60 ended up being resolved by an arbitrated settlement, with 9 others reaching no conclusion, usually because of violent disruption. (William Ian Miller)
So it would seem that even with a solid court structure, a set of laws agreed upon by the members of the lögretta, and even the ability to change your goðorð effectively at will, most cases resulted in violence. A darkly comic section of Njal's Saga deals with a series of revenge killings where Njal and Gunnar effectively pass the same bag of silver back and forth, paying off each successive wergild. So even in situations where a settlement had been agreed upon, there could be extra judicial measures taken. Friedman, though, points out that:
Whether the Icelandic institutions did work well is a matter of controversy; the sagas are perceived by many as portraying an essentially violent and unjust society, tormented by constant feuding. It is difficult to tell whether such judgements are correct. Most of the sagas were written down during or after the Sturlung period, the final violent breakdown of the Icelandic system in the thirteenth century. Their authors may have projected elements of what they saw around them on the earlier periods they described.
So perhaps the system worked better than the sagas would indicate. To put things in perspective, the Sturlung period was about 40 years long. Basically the Sturlung family and a couple others managed to gather a bunch of power and the Norwegian king provided assistance to various groups. Effectively it destabilized the entire Island and can be thought of as more or less a civil war. In those 40 years about 350 people (out of a population of 70,000) were killed in direct relation to this "civil war." And this was considered an exceptionally violent period.
So yeah. Not an ideal An-Cap society, but has some striking features that you would expect to find in one form or another in a true An-Cap world. Even lasted for quite awhile. There is some debate of the efficacy of the system, but with our sources there's no way to know for sure exactly how well it all really worked. Or really if it was even in that exact form.
Footnote