I am not a historian, but I am an epidemiologist, and I think I can adequately address this question without the specifics of 6th century England. Mods, I hope this is ok!

Life expectancy means basically, "Given that you are as old as you are now, how many more years are you probably going to live?" Typically when we say "life expectancy" we really mean "life expectancy at birth," or, how many more years we would expect somebody to live given that they are zero years old.

Life expectancy is calculated based on life tables. Life tables use either data on a population of people of different ages at the same point in time (a period life table), or data on a group of people that is followed from birth until death (a cohort life table), to estimate age-specific mortality. The life table sets age groups (like, 0-1 years, 1-5 years, 5-10 years, ... 80-85 years, etc), counts how many people are alive at that age, counts how many people die before they reach the next age group, and figures out how likely it is for people to die within each age group. Then, you can use those age-specific mortality rates to figure out how many years someone has left to live by calculating out the probabilities. You can kind of think about it like flipping a coin over and over. As long as you keep getting heads, you can keep flipping. The number of times you flip the coin before you get a tails is the amount longer you are expected to live. The important thing to keep in mind is that because we are working with probabilities, the number we are computing isn't an exact figure for "this person will live exactly this many years." Rather, it is an expected value. Half the time, the person would live longer than our expected value, and half the time they would die first.

One of the properties of life expectancy is that the older the age is you count life expectancy from, the older the expected age at death is. Maybe the life expectancy for a 40 year old person is 80 years old. But once that person has reached 60 years old, there is no longer any possibility that they will die between the years of 40 and 60. So now instead of expecting them to live to 80, we might expect them to live to 85. When they were 40 years old, they had to flip the coin 40 times to hit 80, but now they only have to flip it 20 times. Much easier. They can hit 80 and keep on flipping.

With life expectancy at birth, this effect is magnified, because the probability of dying between the ages of 0 and 1 is usually much higher than the years that follow. Here's a current life table for the USA. The probability of dying between ages 0 and 1 is 0.006, after which it immediately plummets: between age 10 and 11, the probability of death is only 0.000097! In fact, the first year of life is the most "dangerous" until you reach your 80s! And that's for the contemporary United States, where we have much better neonatal care than was available in 6th century England.

So: that's why people talk about infant mortality being so influential for life expectancy. But the thing is, there's really not a whole lot of difference between "life expectancy at birth" and "median age at death." Median age at death doesn't mean that most people died between the ages of 25 and 30. It means that half of people died before that age, and half of people died after that age. That's essentially what life expectancy at birth also calculates, albeit through a different technique. Either measure would wind up being influenced by infant mortality. If a lot of people are dying between ages 0 and 1, they count within the half of people who died before reaching age 25. There's not necessarily anything to reconcile.