Or is it complete bullshit which is what I think.
You might be interested in this paper, which reviews life expectancy across different types of cultures around the world.
As you note, infant and child mortality take a heavy toll on average life expectancy. Among hunter-gatherers, only 57 percent of children live to age 15. Of those who reach that age, 64 percent can expect to reach age 45. At that age, the mean life expectancy is about 21 years.
So of 100 hunter-gatherers (on average), 57 live to age 15 and 36 live to age 45. 18 of those hunter-gatherers will live to 66.
So while the odds are against any one person living to become an elder, elders were a fairly common sight even in smaller communities.