There were several large fleet against fleet naval clashes in late Viking times.

These battles were fought by boarding the enemy ships and fighting with spears, swords and shields.

In the Battle of Nisa, in 1062, the forces of Norwegian King Harald Hardradea with 150 ships fought the fleet of Sweyn Estirdsson, King of Denmark with 300 ships. The Battle was won by the outnumbered Norwegians.

The Battle of the Helgea, in 1026, pitted the 600 ship Danish and English fleet of Cnut the Great against 480 Norwegian and Swedish ships. It was a hard fought battle but Cnut won.

The semi legendary Battle of Hjorungavagr, circa 986, was fought between the 180 ships of Haakon Sigurdsson of Norway and the Jomsvikings with 60 ships. The Norwegians won.

The sagas do not relate the number of ships in the battle of Nesjar in the Oslo fjord in 1016 between Olaf Haraldsson (later St. Olaf) fighting to become King of Norway, and the allies of Sveinn Hakonarson. It is thought to have been a fierce battle, with Olaf prevailing.

In 999 or 1000 the Battle of Svolder was fought between King Olaf Tryggvason of Norway and a number of his enemies. Olaf had only 11 ships, while the coalition of Swedish, Danish, and other Norwegian allies had more than 70. Olaf was defeated and died in this battle.

Source:

http://en.wikipedia.org/wiki/Category:Naval_battles_involving

Medieval Europe is a long time so there were definitely changes over time. There were also big differences depending on where you were.

Before the invention of cannon naval warfare was pretty much just an extension of land warfare. You put soldiers on your ship and got close enough for them to shoot arrows and other missiles at the enemy and then got close enough to board and fight them hand to hand. Also it's main mission was usually just to transport troops.

The types of ships you could use depended on if you were in the mediterranean or the Atlantic or north seas.

In the mediterranean the galley was king from ancient times though the end of middle ages. The mediterranean didn't have reliably high speed winds enough to make sailing ships fast enough to compete with oared ships. The techniques used were pretty much unchanged from ancient times. You sought to ram the enemy on the side and then board them with marines. Later on they started to put catapults and then cannons on the galleys which allowed them to attack fortresses and sink enemy ships from a distance.

In the open sea galleys and oared ships in general weren't so useful because the seas were much rougher and the winds higher. Galleys have to have lower sides so the oars can reach the water. This means in rough weather they get flooded and sunk more easily. That being said the warfare techniques were still much the same. Shoot at them then ram them and board them. The sailing ships of the time however weren't much more up to the task than the oared ships. Compared to later sailing ships they were slow, poorly maneuverable and could not sail upwind at all. Consequently overseas trading and the navy needed to protect the trade routes was small to non-existent.