Other examples of it as a luxury item come from various bartending manuals I’ve encountered through research for work— Jerry Thomas, The Savoy hotel bar manual, etc.
Even as late as the 50’s, my mother, who grew up in Michigan, tells me stories of her childhood where oranges and tangerines were used as stocking stuffers, rare treats. I tried googling the inflation for oranges and then tropical fruit in the US and turned up little I could access.
Oranges were not actually rare and expensive commodities in the Victorian era. This is something I touched on once in an answer about A Christmas Carol, but I'm happy to go into more detail here.
The oranges we typically eat today are sweet oranges, previously called "China oranges" as they were originally(?) grown in China and spread west by Arabic groups, eventually coming to grow along the northern Mediterranean. Bitter "Seville oranges", on the other hand, were brought along the southern Mediterranean and eventually up to the Iberian peninsula. By the eighteenth century they were being planted in the Americas and Portuguese island colonies, where climate allowed; in the nineteenth century, China oranges became the dominant type eaten and were what people speaking English meant by an undistinguished "orange".
Certainly by the mid-seventeenth century, both types of oranges were commonly being sold cheaply by "orange-girls" on the streets and in the theaters of London, after having been brought by ships from Spain and Portugal. Nell Gwyn, Charles II's most famous mistress, was originally an orange girl, which became part of her rags-to-riches mystique. The Earl of Rochester wrote a "Panegyric to Nelly" that took her early career as a subject:
The Orange Basket her fair Arm did suit,
Laden with Pippins and Hesperian Fruit,
This first Step rais' d to the wond'ring Pit she sold
The lovely Fruit smiling with Streaks of Gold
Like many young urban women who sold items of little value and therefore made little money, orange girls were understood to be open to selling sex as well as their wares, and they also often acted as intermediaries from more dedicated sex workers and their customers for coin as well.
As I noted in that earlier answer, nineteenth-century English textbooks using fruit prices in math problems put oranges on parity with apples, and this was true even in the United States, where they "may be procured at little more cost than the commonest of our domestic fruits in the more northern states". Citrus fruits' thick rinds allowed them to be transported far without refrigeration, which is why they could be so cheap even before steamships and railroads (and why the British navy could keep so many limes on hand for its sailors on long voyages) - but it's likely that those advances were still welcomed, and brought an increase in volume of shipped fruit.
More relevant to your question, it's likely that these developments also saw imported oranges and other citrus fruits becoming more available outside of urban areas. It's notable to me that most of the information on oranges in England relates specifically to large cities - ships coming in to London from Spain, theaters and street vendors - and I can find very little reference to their availability in rural small towns. Oranges were associated with Christmas in a general way (such as with the "feast" referenced in the Christmas Carol answer, or the chocolate oranges that are still mostly sold around Christmas today) - "Christmas Day without oranges would have been unconstitutional" - for many Americans they seem to have been only available as a treat at that time of year, perhaps because they could be transported farther in the cold weather. By the twentieth century, though, and certainly by the 1950s, oranges being rare treats would have to do with something other than any practical concerns. They were sold in supermarkets year-round, with different varieties from different sources being available at different times of the year.
Much of this was written with my own general knowledge from extensive previous research, but I also referred to/quoted from:
Anthony McEnery and Helen Baker, Corpus Linguistics and 17th-Century Prostitution: Computational Linguistics and History (Bloomsbury, 2016)
Andrea Broomfield, Food and Cooking in Victorian England: A History (Greenwood, 2007)
Dehard Bruce Johnson, Marketing Charges for Oranges Sold in Pittsburgh and Cleveland, 1949-50 Season (U.S. Department of Agriculture, 1953)
"The Orange-Tree", in The New Monthly Family Magazine (January 1850, vol. 2 no. 1)
"Pollen and Honey" by Eugene Wood, in Boy's Life (December 1917)
To piggyback on this question-
My grandma was born and raised in Nebraska during the depression and would tell us this story about how they got one orange on Christmas every year and it was the most delicious and decadent thing they ate all year.
Was this situation specific to her family or were oranges rare and expensive in the Midwest during that timeframe?