Are you looking for the name used before the invasions or after, or just the "viking" word. In ancient Icelandic it was called Bretland but disclaimer: I'm not certain if that is the proper word the Vikings themselves used so wait for an expert to post on that.

After the Vikings established colonies the part of England they controlled was known as the Danelaw. Is this what you are looking for?

Swedish linguistics major here. There are a number of rune stones here than mention warriors that left for or perished in England (written "IKLAT" due to spelling conventions), but I'm not sure what they'd have called the whole island. Remember that Britain in those days was not considered one realm as it is today, like Sweden. Sweden wasn't united until later - under Birger Jarl, as I've been taught. If Roman expression "Britannia" was used in Viking times, at least it doesn't seem to have been picked up by the Danes. I have never seen it in Scandinavian sources.

Well in Icelandic (the closest language to Old Norse) the word for "Britain" (United Kingdom) is "Bretland" other forms of the word Bretland include; Bretlandi & Bretlands, depending on whether you're using the article or not.

The Nefnifall (Nominative) form of Bretland is just simply "Bretland". The Þolfall (Accusative) form of Bretland is also just "Bretland". The Þágufall (Dative) form of Bretland is "Bretlandi". The Eignarfall (Genitive) form of Bretland is "Bretlands". Information taken from the Icelandic language Wiktionary, in their singular forms.

The Icelandic word for "British" is "Bretlandseyjar" The Nominative plural form of "Bretlandseyjar" is simply "Bretlandseyjar". The Accusative plural form of "Bretlandseyjar" is also "Bretlandseyjar". The Dative plural form of "Bretlandseyjar" is "Bretlandseyjum" & the genitive plural form "Bretlandseyjar" is "*Bretlandseyja *". Taken from Bretlandseyjar on Wiktionary.

The "British Isles" in Icelandic would be "Bretlands-og-eyjar". (Britain and Islands)

I'm not entirely sure if the vikings gave Britain & the British another name on arrival though. I do know that "Danelaw" (Danelög in Icelandic) is the area of Britain controlled by the Danish Vikings.

And in a follow up question what did they call or write about Scotland, Ireland and the Isle of Man?