I'm very interested in how estimates of the populations of ancient cities are established, particularly for cities of the prehistoric or protohistoric era, such as Çatalhöyük in Turkey (which Wikipedia lists as having a population of ~1000 c. 7000 BCE) or Uruk in modern Iraq (which has been estimated to have ~45,000 residents around 3000 BCE, to cite Wikipedia again). One factor that seems like it should matter would be the productivity of agricultural or pastoral work undertaken by the residents, and another would be the number of homes discovered in a particular layer. But, even if we can estimate productivity, how do we know how many individuals lived in a given domicile? How do we account for people living in structures made of materials that weren't easily preserved, such as wood, thatch or other organic matter? And how are these factors collated to give a single estimate of population size?
I mean, you’ve basically hit the nail on the head already.
When archaeologists are estimating population size, it’s always going to be a range. So the agricultural or foraging resources available might give you an upper limit (even though archaeologists often overlooked the productive value of certain types of agriculture in the past because they didn’t conform to a western idea of what a “field” should be). On the other hand, you could take as many domestic structures as you can identify,assign a believably low number of people per house, and get the lower limit.
Of course, there’s all kinds of problems with these methodologies. As you said, how do we figure out how many people lived in a house. One way might be with the direct historical approach, where you get as close as you can in time and work backwards from there. (This obviously works better for late Precontact Native Americans than for Catalhoyuk, for example.) Sometimes, you can identify an area that you think is for sleeping (as opposed to food processing, tool production, etc) and estimate how many people could comfortably fit there.
There’s also a few issues that you have to think critically about here. One is, archaeologists until very recently were usually mostly men. As a result, reconstructions of populations have historically tended to exclude women and children, implicitly using some kind of generic masculine individual to stand in for everyone. Likewise, what is comfortable space per person is culturally variable. Even different contemporary industrial societies have different standards of personal space. Surely, people in the past did as well.
The history of increasingly better population estimates is archaeologists developing better methodologies and theory to address the potential sources of uncertainty that I’ve outlined above.