There have been many periods in history where calendars were hard to keep in a reliable way and many people in the past never really paid attention to what year it was. How do we know when the Roman Empire actually existed to a specific year? Or any other historically significant event?
Hi there -- not discouraging further responses, but this section of our FAQ on calendars and timekeeping has a bunch of answers you may be interested in. The short version is that we can be confident that we're not "missing time" due to comparing different calendaring systems with one another, comparing calculations for the date of Easter with one another, and by using astronomical evidence (comets, etc.) to confirm dating of events.