If you're writing about the Viking Age, then we really have very few sources about what these people called themselves. By the mid-800s, there seems to have been a people living in modern Russia/Ukraine who were called Rhos according to Latin sources, Rhos (‘Ρῶς) according to Greek sources, and Rūs (روس) according to Arabic sources. You'll sometimes also encounter the term Rus' in scholarship, which is the English equivalent of the Slavic Русь.

The term Varangian seems to have appeared slightly later in the 900s, typically in contexts associated only with Norse mercenaries in the service of the Byzantine emperor. You should also be aware of the term Garðar, which was an Old Norse term for Russia that appears, for example, in medieval Icelandic texts, as well as Gaill ("foreigner"), which appears in Irish sources in various compounds. The term vikingr does appear in Old English sources, but Anglo-Saxons used this word only to describe pirates and never to describe Scandinavian culture more broadly. Of all these options, I personally prefer the term Rus, which seems to be how the people of the time identified themselves to outsiders.

Of course, we also need to be careful about who we think these people were. There were vikings (i.e. raiders) who went eastwards, but this is probably too narrow a meaning for what you want to investigate. There were also Scandinavians—probably from Sweden—who migrated into the areas of modern Russia and Ukraine. It's not necessarily fair to describe these migrants as Scandinavians, since they didn't have our modern concept of Scandinavia = Denmark + Norway + Sweden, nor would they even have thought of smaller regions like Sweden as a unified cultural area. Political unity only began to develop over the course of the Viking Age, but it was contested and not always along modern political boundaries. Moreover, it's hard to say if family ties to Scandinavia would have remained important for migrants to Russia or their children.

We also know very little about how cultural connections were maintained throughout the Viking Age—Did people from Scandinavia regularly travel to Russia? Did people from Russia regularly travel to Scandinavia? Or was there a third "homeless" group of merchants and adventurers that tied these regions together? Probably a balance of all three. (And which group indeed would someone like the sometime-exile, sometime-usurper Harald Hardrada fall into?)

Again, my personal take is to use the term Norse. No term is perfect and this one has its flaws, but it also avoids some of the biggest pitfalls of other common alternatives. Again, people whose families dwelt in Russia for generations can't rightly be called Scandinavians. Some of these people were at times vikings, but most vikings spent their lives doing other things as well and wouldn't have thought of themselves as belonging to some interregional "Viking" culture. We also know that a lot of people migrated into or took on aspects of the culture we associate with vikings today. A key component of this seems to have been language (Norse), which might well have been the only cultural trait that seal hunters in Greenland shared with the Varangian mercenaries of the Mediterranean.

I consider the sources below to be essential. Franklin and Shepard strike me as the most thorough and reliable, probably the best place to start. It's a great reference. Duczko makes important use of archaeological evidence, although this might be difficult to follow without first reading Franklin and Shepard. Montgomery is a good rapid guide through little-used Arabic sources, and you might be interested in the other chapters on Russia and the East in that same volume. For the Arabic sources themselves, see the excellent collection by Lunde and Stone, although because it's organized thematically, it's easy to lose track of which author is responsible for which passages. I've also pointed to two important works by Abrams and Jesch that might be useful for thinking about who these people were, how they thought of themselves, and how and why they sometimes expressed a "Norse" identity.

Lesley Abrams, "Diaspora and Identity in the Viking Age," Early Medieval Europe 20, no. 1 (2012): 17–38.

Wladyslaw Duczko, Viking Rus: Studies on the Presence of Scandinavians in Eastern Europe, Northern World 12 (Leiden: Brill, 2004).

Simon Franklin and Jonathan Shepard, The Emergence of Rus: 750–1200, Longman History of Russia (London: Longman, 1996).

Judith Jesch, The Viking Diaspora, (London: Routledge, 2015).

Paul Lunde and Caroline Stone, eds., Ibn Fadlān and the Land of Darkness: Arab Travellers in the Far North (London: Penguin, 2012).

James E. Montgomery, "Arabic Sources on the Vikings," in The Viking World, edited by Stefan Brink and Neil Price (London: Routledge, 2008), 550–61.