This article is what you are after.

They have taken samples of people from different periods of history who survived to adulthood(over 15 years) whose life span is known and who didn't die by violence.

In their data

Kings of Judah(1000–6000 BC)lived to be about 52

Greek philosophers, poets and politicians(450–150 BC) to 68

Greeks post 100 BC, 71.5

Roman philosophers, poets and politicians(30 BC– 120 AD), 56.2

Christian Church Fathers(150–400 AD), 63.4

Italian painters(1300–1570 AD), 62.7

Italian philosophers(1300–1600 AD), 68.9

Monks Roll of Fellow of the Royal College of Physicians(1500–1640 AD), 67

Then

1720–1800 AD 62.8

1800–1840 AD 71.2

1931 66.2 1951 68.9 1981 72.0

The older numbers are quite small sample sizes and possibly not representative of the population(as the people whose lifespan we know are almost by definition a privileged group) but generally speaking it seems 65 to 70 is the norm. Although Romans around 1000 years ago were looking at around 56 years which the authors attribute to lead pipes.

You're right though that the really low average life expectancies we see historically(and in developing nations) are due mainly to high rates of infant mortality having a disproportionate effect on the average but improvements in midwifery and obstetrics have dramatically lowered deaths among young women and the advent of hygiene and antibiotics help keep people alive across their lifespan. More recently advances in the treatment of ischemic heart disease have made life expectancy even better still.

It's also worth noting that although most wealthy countries now have a life expectancy of a smidge over 80 that doesn't mean that an 81 year old is due to die, if you make it to 80 you have a good chance of making it to 90. As your question implies using a single number for life expectancy is really problematic in a lot of ways.

Currently it seems like we are pushing up against the boundaries of human lifespan, if nothing else kills you first it seems old age gets most of us in our 80s or 90 with a very lucky, very few making it to the 120s.