You're definitely right about infant mortality skewing the results, but the data you heard isn't too far off for an average person's life expectancy. According to Roberta Gilchrist in "Medieval Life: Archaeology and the Life Course" if a Medieval man in Halesowen, England survived past the age of 20, in the early 14th century his further life expectancy was another 21-28 years, up to fifty so not great, but by the 15th century that was up to 33 years, past fifty and this is for an average for a town. Aristocrats and monarchs might occasionally live longer, John of Gaunt died at 58, Humphrey, Duke of Gloucester died at 56, Edward III died at 64 and Edward I died at 68. Later on in the Early Modern it was longer; Elizabeth I died at 69 (nice), William Cecil 1st, Baron Burghley died at 77 and Charles Howard, 1st Earl of Nottingham died at around 88.

So the data that occasionally come up saying the general average life expectancy was something like 30 is misleading, but 40-50 as an average for Medieval is probably about right.