While it's difficult to grasp what your average peasant would or wouldn't know, it strikes me as unlikely that they would forget who the Romans were. For one thing, the Romans feature fairly prominently in the Gospels, which as I'm sure you can imagine were pretty darn important and widely disseminated during the Middle Ages. Any educated person would certainly know the Romans, most of the Church Fathers wrote within the Empire, and even those who wrote after the fall of the Western Empire wrote amidst its ruins were heavily influenced by their classical heritage (Boethius was a senator and consul from an ancient family, Gregory the Great likewise was from the Senatorial class, for example). Beyond theological works, much of the education curriculum came from Roman sources, Cicero, Donatus, etc. Horace, Ovid, and Virgil were all still read, Virgil especially was tremendously respected. They also were aware of basic Roman history, which featured heavily in encyclopedias and historical works. References to Greco-Roman mythology pepper sermons, theology, and art. I think it's reasonable to assume that this knowledge trickled down, at least somewhat, to the lower illiterate classes.

edit: Stupid of me not to mention at first, but the Roman Empire also still existed until the 15th century, and they employed Norsemen as mercenaries from the 10th century onward.

It's by no means definitive, but one interesting Anglo-Saxon perspective on the Romans comes from a poem from the 10th century Exeter Book called 'The Ruin'.

The poem is an elegy, which describes a ruined Roman city from the perspective of an Anglo-Saxon in the 7th-8th century. The city mentioned is probably Bath, though it's been argued that the site may have been at Hadrians wall. The poem gives a us a sense of a colossal, almost mythical civilization that has collapsed and fallen into ruin.

 

The opening lines actually mention giants:

 

Wondrous is this wall-stead, wasted by fate.

Battlements broken, giant’s work shattered.

Roofs are in ruin, towers destroyed,

Broken the barred gate, rime on the plaster,

walls gape, torn up, destroyed,

consumed by age. Earth-grip holds

the proud builders, departed, long lost,

and the hard grasp of the grave, until a hundred generations

of people have passed.

 

The poem gives a sense of an advanced civilization, poorly matched by kingdoms of the present day (this has actually been a common trope in fantasy fiction since Tolkien. In fact, Tolkien was a renowned Anglo-Saxon scholar and he may have gotten some of his inspiration for the Lord of the Rings from this very poem):

 

Decay has brought it down,

broken it to rubble. Where once many a warrior,

high of heart, gold-bright, gleaming in splendour,

proud and wine-flushed, shone in armour,

looked on a treasure of silver, on precious gems,

on riches of pearl...

in that bright city of broad rule.

 

There's also a reference to Roman building methods:

 

Mood quickened mind, and the mason,

skilled in round-building, bound the wall-base,

wondrously with iron.

 

This probably refers to Roman Concrete reinforced with iron bars. The Romans used an inferior, rubble cored form of concrete in Britain (in the absence of volcanic ash). This was a building method which the 8th century natives neither understood nor had access to. In fact, the secret of Roman concrete was lost altogether for another thousand years.

Living in a post Roman world must have been somewhat strange for the later Anglo-Saxons - encountering these kinds of fantastic ruins on their travels, farming under the shadow of ruined aqueducts and impossibly tall city walls, occasionally unearthing Roman treasures and coins from far-off lands. They may not have seen the Romans as giants, but the average Anglo-Saxon was most likely aware that a great empire once ruled the island and had passed on.

(The translation I used is here and a second, longer-form translation can be found here.)

Alfred the Great, as in the first King of a United England, studied in Rome for uears, he and his predecessors absolutely knew about the romans.The Priests across Britain were educated and many of the more important Kings and Southern Kings were very well educated, in Wessex probably to the level of France or Bavaria.

So whilst the odd peasant might not have known, Lords across Britain would definitely know. Remember, theyd only been gone 400 years.

By contrast, the Saxons in Bernard Cornwell's 'Saxon Stories' series pretty much seem to know exactly who and what the Romans were (people from Rome who made a great empire, and built all their bridges, basically), though they don't know why they fell, and they also seem to have no clue that the Eastern Roman Empire still actually existed. Is that more accurate?

In more detail, The Saxon's Stories feature the life and achievements of Alfred the Great. Would have he known that the Roman Empire still endured?

From what I remember watching the show, I get the perspective that the Saxons were more referring to the Fomorians & the Saxon king Ragnar was talking to disputed their existence by pointing out Roman relics, stating these "predecessors" must have been human.

side question to OP: Which episode is this mentioned? I never followed it cloasely, but this sounds like an episode I'd like.