Did any countries on mainland Europe ever consider invading Viking territory in Scandinavia?

One might see the entire Frankish expansion northward as an attempt to stop piracy and robbery, including the wars with the Frisians in the early 8th century and the Saxon Wars of the late 8th. By the time the Franks had reached modern Denmark, the Viking Age as it is traditionally known had begun, but one should keep in mind that raids from the North on the modern-day Netherlands already took place earlier as well. From a Frankish perspective, Vikings were not necessarily anything special, but piracy on the North Sea was something that they had been dealing with for centuries (actually continuously since Roman times). Internal Frankish trouble in the 9th century did halt expansion northward for a while, but the Ottonians (German East Frankish kings) in the 10th century continued the process, contributing to the Christianisation of Scandinavia in the second half of the century (and possible victory over Denmark, but this is uncertain). So yes, continental powers did invade Viking territory, but only those areas that were within reach. This completely makes sense from a Frankish ruler's perspective: why would you want to wage war somewhere against some peasants with boats in the far north, when you can fight campaigns against much more valuable targets (Italy, the Avars, fellow Franks, Iberians) where you can gain the revenue to actually pay your followers. A much more efficient strategy for securing your coasts is just fortifying key points, bribing the attackers, or asking the attackers to take care of the affected area and protect it against other Vikings. These are the strategies that the continentals used when the Viking attacks became too severe, rather than building and equipping a fleet and going house-to-house in Norway, as the Romans might have done (and unsuccesfully did around the turn of the millennium), after which they adopted the more sensible divide-and-conquer policy of befriending allied kings).

As /u/ritzamitz already suggested, the Slavic (and possibly also the Baltic) peoples of the South Baltic coast also raided Scandinavia (particularly the islands), but I can not justify calling these 'countries'.