I know there was a lot of piracy - eg according to Snorri, pirates from Estonia butchered and enslaved the crew of the ship Olaf Tryggvason was traveling on. (Perhaps I have not looked hard enough, but) I've seen no records of serious naval battles from the Viking era or before, in Northern Europe. When did genuine naval battles start in Northern Europe, and what were they like?
Well, it really depends on your definition of "genuine naval battle." We have records on the use of ships to raid across the North Sea and the Channel coasts dating back to classical antiquity, and it's reasonable to assume it happened even earlier.
But true naval warfare, that is warfare unconnected with war on land, we mostly date to the invention of cannons that allowed ships to attack one another without grappling (that is, essentially boarding another ship and fighting a land battle on the ship). I wrote an answer here about naval warfare before the invention of cannon that may be of interest:
To quote from that a bit:
What we know about naval warfare in and around Britain in, say, the period of the Norse invasions, is that ships were primarily used to move troops and their equipment around. There was no true naval warfare, if by naval warfare we mean warfare unconnected to war on land. Rather, ships were used to project power on the land, and the seas remained a debatable place. That meant that, on the one hand, there was little one could do to prevent an enemy landing troops on one's land, but on the other hand, one could equally as well count on using one's own ships and forces to force a landing on an enemy's coast. The scouting systems used at sea that would prevent that didn't develop before the 16th century or so.
When we see accounts of naval battles, they are universally fought inshore, usually in a bay, an estuary or even in larger rivers, and are usually fought as an auxiliary to a land warfare engagement. There were no missile weapons that could sink ships, and it wasn't possible to fit rams to the ships of the Norse and their imitators, so it was only inshore that grappling and hand-to-hand fighting could take place. Also, inshore communications were quick enough that they could on occasion call for help when a landing occurred, a prerequisite of getting ships together to fight.
Naval warfare in this era has been compared to mounted warfare on land: ships were a way to transport a group of raiders quickly to a destination and achieve strategic surprise. Shallow-draft ships could also carry raiders hundreds of miles into the interior of a country, using rivers for transport. A scenario might be that a group of raiders has landed at a village, dragging their ships onto the strand, and that a counter-force manages to trap them with beached ships and burn their means of escape; the raiding force could then be hunted down on land. Many of the battles that we know about during this time period seem to take that form (the details of naval battle are very incomplete). Alternatively, there are some accounts of battles on the water between two fleets, where it seems likely that individual ships grappled with one another, with boarding being the decisive factor in their success. We do know that defense against ships in this era required a combination of fixed defenses (bridges, forts) and squadrons stationed at spots where they could quickly be called to respond. The accounts of battles that we do have tend to list a prince and his achievements (Alfred of Wessex went to sea with a "fleet" (sciphere) in 875, fought seven enemy ships and captured one ... etc.)
EDIT, the First Also, here's another thread that you may find of interest, about naval warfare in the Middle Ages:
https://www.reddit.com/r/AskHistorians/comments/3mejyz/how_did_navies_operate_in_the_middle_ages/
EDIT, the Second Actually this particular comment from that thread gets to dating the use of cannons:
We know that heavy cannons were almost certainly used on board ship in 1504 and 1506, when James IV was at war with the rebellious Donald Dubh MacDonald, and captured two castles by bombarding them from the sea. The Scottish ship Michael seems to have been the first designed to carry heavy cannons from the beginning.