Nowadays, we have all the fruit we want, whenever we want it, in the supermarket. But did people eat fruits in winter, in countries where it was frequently freezing?
Generally they didn't. Im sure there are some exceptions here and there, but up until very recently the vast majority people ate out of their root cellar until for the cold months.
In the Adventures of Tom Sawyer a wealthy character gives Tom an orange and it is treated as a major luxury. Remember this story took place in Missouri on the Mississippi River, not that far from Orange country.