The Julian Calendar adds a leap day to its 365 day year every 4 years, to bring the average year to 365.25 days
The real year is actually 365.24219 days, meaning that (365.25 - 365.24219 = 0.00781 days = ) roughly 11 minutes were erroneously gained a year, or ( 1/0.00781 =) 1 day was gained every 128 years.
The Gregorian calendar came into effect in 1582, and the Julian calendar in 46 BC, meaning that there are 1,628 years for the Julian calendar to lose sync to the real year.
1628 / 128 = 12.7 days askew
Why did the Gregorian calendar not add 12 or 13 days, but 11?
This question involves a whole bunch of things: not just 15th century astronomy but also 2nd century debates between Christian theologians, 1st century BCE regime changes, and Hellenistic astronomy.
OK, first things first: it was 10 days, not 11. (You're probably thinking of when the Gregorian Calendar was adopted in the anglophone world in 1752: at that time the error was indeed 11 days.)
Now, the core of the answer is that 16th century astronomers weren't trying to restore the calendar date to the point in the solar year that it was at in 46 BCE. They were trying to restore it to where it was in 325 CE. No one cared about Julius Caesar. But the Church did care a great deal about how to calculate the calendar date of Easter. And Easter is a messy business because it's affected by both the lunar calendar (which determined the date of Passover in antiquity; nowadays it's lunisolar) and the solar calendar (which was used in the eastern Roman empire after 30 BCE or so, when Egypt adopted a solar calendar following the end of the Ptolemaic dynasty).
Because of that messiness it took several centuries for early Christians to come to an agreement on when to celebrate Easter. The debate began in the 150s CE, between the Anatolian and the Roman branches of the Church, and it wasn't finally agreed until the Council of Nicaea in 325. They came up with a hybrid formula, and people are still unhappy with it because it's still messy. But since that's when the date of Easter was formalised, that's the point in the history of the solar calendar that everyone cared about.
There's one other small misassumption in your question: the real year is indeed 365.24219 days, but 16th century astronomers calculated it as about 365.24254 days. The actual calendar as adopted, the Gregorian Calendar, is 365.2425 days, exactly 3 days short of the Julian Calendar every 400 years. So that's the rate that was used for calculating the drift.
Plug the revised date and number of days in, round up, and you get 10 days: (1582-325)*3/400 = 9.4275, round up to 10. Do the calculation again for 1752, which is when the English finally adopted the improved calendar, and you get 10.7025, which rounds up to 11.
Now, an endnote to consider an alternate history: another possible course open to them would have been to align the Gregorian Calendar with the traditional dates of the solstices and equinoxes. The traditional date of the spring equinox was on 25 March (and this too played a role in assigning the date of Easter), with Christmas on 25 December on the traditional date of the winter solstice.
But I'm pretty sure they didn't have the means in the 15th century to work that out: the equinox last fell on that date a few centuries even before the Julian Calendar was implemented, so we're talking a retrojected Julian Calendar. Even in 46 BCE, when the solar calendar was implemented, it was noticed that the equinoxes had drifted relative to the 365.25 day cycle, compared with the time when the 365.25 day cycle had been calculated. That allows us to conclude that the traditional placement of the equinoxes and solstices in the 365.25 day cycle was calculated around 400-300 BCE.