Well the Kings' Sagas are generally viewed as little more than legend rather than historical fact and this would not least apply to the Saga of Saint Olaf, since Olaf was very popular and had a ton of legends about him. Add to that that the context there is Olaf on a general tour of battling his way around the Baltic, going to Sweden and supposedly creating the southern (Söderström) channel around Stockholm (Agnefit) out to the Baltic by digging, and fighting in Estonia, Denmark, Frisia and other places besides Finland. It's likely to reflect folklore about Olaf but is not thought to have much historicity. I'd say it's not so detailed either; the prose part of the story, like much Snorri wrote, is likely to mainly be an extrapolation of the poetry part (which are generally considered to be older)

These stories of Sweden and the East Baltic are possibly of the same likely-Swedish origin as the Ynglinga Saga in the Heimskringla, which also has several stories about Swedish-Finnish interactions, including both raiding and intermarriage. One of which is the story of Agne who is deceived and killed by his Finnish bride as revenge for the murder of her father; Agne becomes the namesake of the village Agnefit where Stockholm now is and featured in Olaf's Saga. (Olaf also fights Sóti in Sweden, who appears to give his name to Sótasker, so another connection between the stories here in the Swedish place-name origins)

But while the particulars of the story are unlikely to be true, the story itself and the repeated motifs do tell us something about how 13th century and likely earlier Scandinavians - or at least the Swedes - viewed Finns. It's likely that Finns would appear more in Swedish stories as not only is Sweden much closer to Finland than Denmark or Norway, but Swedish Vikings also sailed east to a greater extent, into the Gulf of Finland and up the Neva river. There are many archaeological Viking finds in that area as well. In Estonia the Salme ships show that raiding in the Baltic region preceded the Viking Age proper, defined by farther-away raids in England and continental Europe.

So both the sagas and archaeological evidence show that there were Finnish raids going on, but not only in one direction; for instance the pillaging of Sigtuna in Sweden in 1187 (albeit a century after the Viking Age) was supposedly done by Karelians. But since the Ynglinga Saga and Hervarar Saga speak of intermarriage in several cases (unhappily as they may be) it seems entirely plausible that such things happened. Finns are depicted as others but not entirely foreign. Another recurring theme is their supposed skill at magic/witchcraft, a stereotype which would live on well into the early modern era, particularly about the 'Forest Finns' of Sweden.

Written sources from the Viking Age in Scandinavia simply don't exist, but even from the 12th century essentially nothing is known; supposedly Saint Erik of Sweden launches his crusade to christen Finland in this century (written down only around the year 1300 though) but even this - centuries later than Olaf - is commonly doubted and the only more definite accounts puts Swedish conquest of Finland to the early-mid 13th century. (Runestone G319 in Gotland from that period also speaks of one who died in Finland) The Novgorodian Chronicle also mentions Swedes attacking together with a number of Finnic tribes from the Gulf of Finland area in 1240, although there is no Swedish record of it.

Linguistics is often a source for the Viking Age as we have no written sources, and it gives perhaps a more objective testament to relations between Scandinavia and Finland, in that Finnish borrows a number of not only Norse words but proto-Norse and even Proto-Germanic. (e.g. Finnish kuningas - king - is more similar to reconstructed proto-Germanic 'kuningaz' than the decendants, Old Norse 'konungr' and Swedish 'konung', or juusto - cheese - vs proto-Germanic 'justaz', Old Norse 'ostr', Scandinavian 'ost') This would indicate that close contacts preceded the Viking Age by centuries.

There are inland finds from Finland that seem to show Scandinavian contacts far beyond coastal regions; such as in Tuukkala near Mikkeli, where a 11th century brooch from Gotland. (FiSAO1987;38) was found. Even as far off as Salla, in northern Finland near the Russian border, there's been finds of hoards of coins that include for instance Anglo-Saxon coins of the Viking Age. Yet at the same time Finnish coin hoards have a different composition of coins than Scandinavian ones, with a larger proportion of coins from Germany. Which both serves to show that the Finns were part of the Scandinavian/Baltic trade network, but at the same time may be evidence that the Finns were also acquiring coins through their own independent travel and trade.

So it's a mixed bag. Relations go very far back and were neither uniformly hostile but not entirely peaceful either. (But for that matter, neither were relations between Scandinavians) Primarily contacts would've been with Swedes and seems to be mainly Swedish-sourced stories from Heimskringla in which they appear, both as allies even if mostly as adversaries - but quite consistently magical and not to be trifled with; conjuring storms and putting curses on their adversaries. Even in the Icelandic Sagas that theme recurs, e.g. Þorkell in Njáls Saga, who goes east with Sverker Karl from Sweden and in Bálagarðssíðu - the Finnish coast between Helsinki and Turku, runs across and slays a monster, the finngálkn. - the name itself possibly another allusion to Finnish magic. I'd say there's an undercurrent of fear and respect in the Sagas. although day-to-day relations were perhaps on the whole much more a matter of more mundane things like trading.