I've explored the issue of circumference in Columbus time few times before.

To summarize, Europeans actually didn't know the exact size of the Earth. They had several values floating around, and none of those were completely accurate (some were close) and what's important none seem to come from personal astronomical observations, but from the interpretation of ancient and medieval geographical and astronomical text and values that could be found there. Particular source of error was lack of references to translate ancient units, like Eratosthenes 240,000 stadia, or Alfraganus' 20,400 Arabic miles to units they themselves were using, usually iberian leagues or italian miles.

Of the guesses of circumference Europeans toyed with, Columbus's was one of them yet one of the lower ones. But it's important to keep in mind that this value wasn't invented by Columbus but he adopted it from more respectable works. Columbus did claim he confirmed his value by measurement, which can't be true as it wasn't a correct value.

It is also true that Iberian cosmographers and scientist disagreed with Columbus on his estimate, and with that in mind your question still applies. How did the cosmographers and astronomers reconcile the new data with old theories?

We will never have definite answers on this, but we have some indications. Following Columbus return from his first voyage, Portugal and Spain entered into a diplomatic dispute over ownership of new lands. In 1494 in treaty of Tordesillas they divided their respective zones by a meridian line 370 leagues west of Cape Verde islands. The negotiations before the treaty and afterwards, to determine where this line actually is, reveal us that there was a considerable debate over the value of circumference (length of one degree). Thus this innocent scientific issue became a political tool, as both sides tried to propose a value which they thought suited them. None seem to accept Columbus value which is interesting. Spanish proposed a value of 24,000 italian miles (16.67 leagues per degree) which is lower then real, but higher then Columbus, while Portuguese proposed still too low but closer to reality value of 25,200 miles (17.5 leagues per degree). I tried to the get to the bottom of why they chose those values in the second link post, and I concluded that these values most likely come again from interpenetration of ancient greek texts (Posidonious and Eratosthenes values in stadia, converted by ratio 10:1). Yet as Portuguese value was a newer proposition, and closer to reality then previous ones, it also might mean they had conducted astronomical experiments or calculations to confirm their values. But we don't have records of any so it is purely conjecture.

In the end, determining circumference isn't as trivial as it seems. A major problem back then was measuring the distance between two points especially when they are more then 60 miles distant. Furthermore, measuring longitude in degrees was impossible back then, and while the astronomers of the time were increasingly accurate in determining latitude there were still errors.

Ultimately, we do have some correct measurements popping up from time to time in history, so we know some ancients were capable for quite correct calculations. The circumference values proposed in Columbus time ranged in error of like +-15 percent from actual value, and that seems like nothing when you are aware of the problems I listed above. Ultimately for Europe, problem would be solved in 17th century with Jean-Félix Picard, and then Isaac Newton who (re)calculated the correct value

EDIT: I made a ridiculous blunder of confusing latitude and longitude properties, which I removed now