That's a fascinating question! And the answer is ... probably not ... although they probably didn't think of the world as round either. Our knowledge about Viking Age worldviews, or more precisely, cosmography comes above all from a collection of Icelandic poetry and an Icelandic poetry handbook both written around 1200. These are commonly referred to as the Poetic and Prose Eddas.

There are problems with seeing these texts as representative of Viking Age beliefs. They were written over a hundred years after the "Viking Age" proper (ca. 750-1050) had ended. They were written in one very small corner of the Norse-speaking world. The people who collected these myths lived in a poor, subsistence economy of farmers and herdsmen, which had very little in common with the booming maritime societies that had made the Norse diaspora possible. Perhaps most importantly, these myths had been preserved for 200 years in a Christian society, which impacted which stories were repeated, how they were adapted and reinterpreted, and which ones were ultimately written down.

We have no way of knowing just how different the myths preserved in the Eddas accurately represent the Viking Age. I am perhaps more skeptical than most, but I think it's worth noting that in those few cases where stories found in Iceland are also found elsewhere in the post-Norse world, they appear radically differently. Character names are often the same or similar, but their relationships to each other get jumbled, and sometimes the good guy in one version of the story appears as the bad guy in another. It's also important to note that archaeological evidence suggests that religious practice (i.e. the things people do based on the things they believe in) similarly differed across time and space, with things like burials that were acceptable in one place being completely out of line in another.

With all these hesitations in mind, the Norse myths describe a cosmography based on several levels connected by a tree. These levels are often described as the nine homeworlds, although the original Norse níu heimar implies a meaning of homesteads and has nothing to do with our idea of worlds as planets. Images like this are especially misleading, since they impose modern ideas of astronomy and the importance of cardinal directions on a mythology that had no interest in these things. Instead, each homestead has was defined simply by its own climate and geography, and most were populated by some form of intelligent life (gods, dwarves, humans, giants).

It's easy to sympathize with this idea of the cosmos if we put ourselves in the place of the early settlers on Iceland. They lived a life defined by archipelagoes—the Norwegian coast, the Faroes, the Orkneys, the Shetlands, Britain and Ireland—each with a distinct geography, climate, and population (although these were admittedly much more fluid than we sometimes pretend). These distinct homelands provided an appealing metaphor for making sense of the larger cosmos.

But the idea that these homesteads were somehow connected by a tree confuses the issue. Were these homesteads hanging from the branches? Were they somehow layered one on top of the other? And what, after all, was the tree standing on?

I don't think these questions would have bothered people living in the Viking Age. They weren't thinking about heaven above and earth down below, and they weren't thinking about planets of gods and giants and men circulating in a single universe. They don't even seem to have thought in two-dimensions and showed absolutely no interest in transforming their lived experiences of the world into a flat and lifeless map. Their geographic descriptions were instead dominated by itineraries and routes. They could describe the route from Bergen to Reykjavik, or from Oslo to Stockholm, without worrying about how these places would appear from a satellite in space. To ask the obvious question, why would they?

The descriptions that survive from the Viking Age instead focus on how geography was experienced on board a ship, which makes a lot more sense. What islands did you need to pass, what side of the ship should they be on, and how close should you get? Even when they used words like north and south, it's evident that they were thinking about these as being directions as relative points along a journey rather than universal directions that covered the entire globe. For example, Icelanders who traveled to Greenland needed to make a long journey west, which probably explains why the first settlement they hit was the East Settlement, and the last settlement was the West Settlement. If you look at a modern map, however, they're clearly north and south, but that's because we're more interested in conceptualizing the whole globe rather than how to get from point A to point B.

All this goes to say that Norse myths don't actually get us terribly close to how Viking Age peoples experienced and understood, but the evidence that does survive more directly from the Viking Age (especially the travelers accounts of Wulfstan and Ohthere, which were written down around 890 in England) show that at least some people's understanding of the geography was strongly shaped by time spent aboard ships. Although many people during the Viking Age lived on farms or herding sheep, it's these people who were connected to the sea who most often attract our interest, and they also seem to have played a dominant role in shaping their Viking Age societies.

So, even though we lack specific information, what can we deduce about their ideas about the world? Well, seafarers are perhaps the persons best suited for observing that the world is round, or at least curved. When a ship comes over the horizon, its top appears first, implying that it's somehow beneath you. The ancient Greeks used this as one basis for deducing that the world was curved.

But the Greeks made the additional assumption that the curved surface of the earth didn't just stop somewhere but made it all the way around, forming a sphere. They had no basis for this assumption (at least not that I know of, although I don't pretend to be an expert in antique thought), although it fit nicely with their ideas about there being perfect shapes and especially the perfection of spheres.

People in the Viking Age, in contrast, showed little interest in symmetry, and perfect circles or spheres are exceedingly rare in the artifacts they left behind. They did mint some round coins (although they hacked a far greater number of these up into small irregular slivers) and they did wear some spherical beads made of clear quartz. The most common round artifacts are loom weights and spindle whorls, both of which served highly functional purposes for textile production and are almost entirely undecorated with very rare exceptions. These do not seem to be guided by their ideas of perfection.

In short, most people in the Viking Age would have experienced the world as a network of routes, and there's no evidence that they ever paused to imagine what these routes would look like from space. Many of them had the opportunity to make the observation that the earth is curved, but with the vast expanses of the North Atlantic on one hand and the endless Asian steppe on the other, there was little reason to think about where the world ended. Some might point out that there is a myth about the Midgard serpent that surrounded the world, but, again, this should be more properly thought of as a homestead in a larger world rather than a self-contained planet. The myths that do survive similarly suggest that Viking Age conceived of these homelands scattered across a cosmography unified by a tree but had no interest in where this tree's world ended. Most importantly, because these people seem to have thought about the world mostly in terms of journeys from point A to point B, and because they didn't value perfect shapes and surfaces like the ancient Greek philosophers, there's little reason to think that they would have moved from the observation that the world is curved to questioning much less assumping that the world might be round.