The population history of Iceland has been intensively studied. The Norse seem to have started settling Iceland in the late 800s. By about 1100, the population had grown to an estimated 70,000 – 80,000 (entirely rural).

http://www.icelandexport.is/english/about_iceland/population_in_iceland/

In 1241 Iceland was struck by its first epidemic of smallpox. This was a catastrophe which knocked the population down to below 40,000. Smallpox epidemics recurred many times in Icelandic history with deleterious effects on population levels. (Smallpox never became endemic in Iceland, remaining an epidemic disease – epidemic diseases are worse for population levels than endemic diseases) (Smallpox epidemics in Iceland are often used to compare / contrast the effect with that of Smallpox on Native American populations after the Colombian exchange) (Population estimates in Iceland are much better than those in the Americas, because the local churches kept fairly reliable records of the Icelandic populations.)

http://www.alternatehistory.com/discussion/archive/index.php/t-211600.html

As well as recurring smallpox epidemics, Iceland was hit with the Black Death in 1402. This smashed the population back down below 40,000 again.

http://www.sagamuseum.is/overview/the-black-death

In 1783, the eruption of the volcano Grimsvotn killed about 25% of the population of Iceland, again knocking the population down well below 40,000.

http://en.wikipedia.org/wiki/List_of_volcanic_eruptions_by_death_toll

The population of Iceland did not ever recover to the 70,000 it had been in circa 1200 until about 1900.

If you are trying to study the demographics of Iceland, it is important to realize that epidemic small pox (with some of the epidemics happening as late as the 18th century), the Black Death, and volcanic catastrophes, kept the population of Iceland well below the carrying capacity of a rural economy well into the industrial period.

On the other hand, the population of Iceland throughout its history is one of the most studied and best estimated populations for the period in the entire world, so with a bit of research and googling, you should be able to find plenty of demographic resources.

In addition to /u/RioAbado's comments, populations have also been generated based on resource availability. See Baumhoff's estimates based on fish availability among Athabaskans in California in "California Athabaskan Groups".

With regard to extrapolations based on skeletal populations, Kenneth Weiss published a number of Life Tables that could be used to project population behavior given, as /u/RioAbado notes, some use of speculative fertility and mortality schedules. See: Weiss, Kenneth M., 1975, Demographic disturbance and the use of life tables in anthropology. In A.C. Swedlund, ed., Population Studies in Archaeology

I can't answer specifically for Iceland, so I hope a specialist can help you there, but I can tell you what the general archaeological approach to demographic reconstruction is when we don't have census data.

There are two components, the architectural and skeletal evidence.

For the architectural evidence, if you excavate a 17th century Icelandic village, for instance, you should be able to determine about how many households are present. Once you know that, you can make some assumptions about average family size and how many of these houses would be occupied at once by how many families. Given those assumptions, you can get a rough estimate of how many people would live in that village. Now, if you have enough archaeological sites from a period you can extrapolate that information to total population.

If you then add in skeletal data, you can often get good measures of average life expectancy and sometimes fertility rate, and so can use that information to supplement the architectural data. If you know about the mortality and fertility rates of a population, and have a reasonable guess at the overall population size from the architectural data, you can extrapolate forwards to create a demographic trend line for a certain period of time as well as having general indicators about health.

I know that isn't what you were asking for specifically, but I hope it at least gives you an idea of one approach to reconstructing demographic information without census records.

Edit: I think the important thing to stress here is that these are all rough estimates. Obviously, the assumptions you make about family size and other factors are going to skew your final counts, but as long as you are careful in doing so you should at least get in the ballpark of what actual population size was.