Was it simply a reaction to shamanism, or were there some misunderstood incidents? Do we know if the Finns, Tavastians, and Karelians held the same view of the Sámi?
While more can always be said on the topic, I hope the following posts respectably by /u/Platypuskeeper and me help to understand the surroundings better:
+++
Well....., it was mainly the Sámi that was called "the Finns" by the Scandinavians especially in western Old Norse texts (authored by the Icelanders and the Norwegians), and this "Finns" were actually the very encompassing concept of distinguish them as the "others" associated with non-Christian magic practices as well as nature, from "us", the Christian-Norse peoples.
There are several alleged witnesses of alleged Sámi's shamanistic and magical practices in medieval Scandinavian texts (especially the detailed ones are relatively less known the Passion of St. Olaf's miracle episode and History of Norway (Historia Norwegie)), so it was clearly not just an unfounded literary prejudice against these "Finn" group(s) of non-Christian, mainly hunter-gathering peoples in the Far North, though.
The following is a famous account of the witness of so-called shamanistic battle between the Sámi magic practitioners in the middle of the 12th century, found in the latter text:
"Their intolerable ungodliness will hardly seem credible nor how much devilish superstition they exercise in the art of magic. For some of them are revered as soothsayers by the foolish multitude because whenever asked they can employ an unclean spirit, which they call a gandus, and make many predictions for many people which later come to pass. By marvellous means they can also draw to themselves objects of desire from distant parts and although far off themselves miraculously bring hidden treasures to light.
Once when some Christians were among the Lapps on a trading trip, they were sitting at table when their hostess suddenly collapsed and died. The Christians were sorely grieved but the Lapps, who were not at all sorrowful, told them that she was not dead but had been snatched away by the gandi of rivals and that they themselves would soon retrieve her. Then a wizard spread out a cloth under which he made himself ready for unholy magic incantations and with hands extended lifted up a small vessel like a sieve, which was covered with images of whales and reindeer with harness and little skis, even a little boat with oars. The devilish gandus would use these means of transport over heights of snow, across slopes of mountains and through depths of lakes. After dancing there for a very long time to endow this equipment with magic power, he at last fell to the ground, as black as an Ethiopian and foaming at the mouth like a madman, then his belly burst and finally with a great cry he gave up the ghost. Then they consulted another man, one highly skilled in the magic art, as to what should be done about the two of them. He went through the same motions but with a different outcome, for the hostess rose up unharmed. And he told them that the dead wizard had perished in the following way: his gandus, in the shape of a whale, was rushing at speed through a certain lake when by evil chance it met an enemy gandus in the shape of sharpened stakes, and these stakes, hidden in the depths of that same lake, pierced its belly, as was evident from the dead wizard in the house.
On another occasion, when Lapps side by side with Christians were trying to hook the squamous flock, the Lapps had noticed creels almost full of fish in the dwellings of the Christians, and these they drew from the water’s depth and almost filled their boat with fish (From: History of Norway, the translation is taken from: [Kunin (trans) 2001: 6f.]."
On the other hand, the non-Christian Sámi people rarely got into the open conflict with the Christian Norse peoples even after the latter's Christianization - they were, so-to-speak, "trading partners" of the animal products like the pelt and fur hunted in the Fenno-Scandia, and the Sámi people also needed the middlemen Norse or other tax-collectors/ merchants from the South who in turn delivered them to the European market, as I also illustrated before in: Did the Sami in the 1400 have freedom? (In Sweden).
The church and other Christian rituals like the Mass also sometimes held in the trading spot between the Norse and the Sámi (though not always - the kind of Sweden prescribed the foundation of the church with such a trading camp first after the Reformation), and once or twice the visit of Sámi sorcerer into the Mass that led not to the apparent conflict is even recorded in medieval texts, as I also cited before: In the Middle Ages, what were relations like between the Sami and Norse?. So, their religious difference (and the "magical" practice of the Sámi people did not generally pose the decisive hindrance in their interaction like trading.
While the most difficult part to give an definitive answer to OP's original question is the general view/ prejudice of other "Finnish" peoples against the Sámi people - since they were also sometimes conflated as the "Finns" with the Sámi by the Scandinavians and didn't have their own written records well into the Middle Ages, it would probably be less problematic for them to align with the Sámi people, at least in regard with the latter's religious practices.
There were certainly a few records of the skirmishes between the Sámi and Norse peoples mainly in Later Middle Ages, and it was perhaps the Karelian merchants or tax-collectors that instigated the former to ambush the latter side's collectors/ merchants, as I also mentioned before in: Did the Sami of Northern Scandinavia ever have any warriors?
These Finnish peoples like "the Kvens (Kvenir)" probably also acted as the middlemen in the Far North fur trade with the Sámi, and sometimes competed not in totally friendly manners with the Scandinavians especially during the Middle Ages - as I explained in cluttered manners before in: What religion did the Kven people practice in the 1300s?
Additional References:
+++