The earliest mention that I can find of oranges in Britain is 1560. Drummond and Wilbraham, in their classic 1939 work The Englishman's Food, describe an Elizabethan dinner with 'a piece of bief, a loyne of veale, 2 chickens , orenges and sauce.' The authors explain that the household in which this was served must have been quite wealthy, for "'orenges' were an expensive luxury. Oranges were sometimes called 'portynggales' (Portuguese oranges) and under this name they appear on the menu of a banquet given by the Skinners' Company in 1560."

So, going from this little snippet in an older but very thorough work, my guess is that oranges were first brought at some point in the medieval period. It's possible they made it during the Roman period, but I'm not at all sure that the Romans cultivated them. Given the original, geographically specific name, the respective climates or Britain and Iberia, and the long history of trade between the two regions, my guess is that as soon as there is regular cultivation of oranges in Iberia, they make their way to Britain as an expensive, exotic treat. Whether that's as far back as the Roman period or later, in the medieval period, I'm not positive.

J. C. Drummond and Anne Wilbraham, The Englishman's Food: Five Centuries of English Diet, page 62.

The OED has citations for orange (the fruit, rather than the colour) from the 15th century onward, including the Paston letters (1470):

Dame Elyzabet Calthorp is a fayir lady and longyth for orangys, thow she be not wyth chyld.

So the fruit was certainly known by then, and apparently sufficiently well-known for a lady to long for them. The Latin word citrangulum was used in British sources from 1250, according to the Revised Medieval Latin Word-List, but that may only indicate knowledge of the fruit, rather than its availability.