So I studied maritime archaeology, not military or naval history - so I have an overview that I learned during my MA. There are many others who would know much more, and in more detail. But I will describe to you what I know...

There are roughly three stages, and they are not necessary a chronological evolution, and you must account for whether the ship is being rowed or sailed:

1. Ship as transport

This means that one ship encountering another goes and finds a bit of land to fight each other. This is not part of my notes where I have no references, so I’m not sure where that comes from. Perhaps someone could elaborate for our benefit.

2. Ship as a fighting platform

This means that the ships meet on the water and fighting takes place on deck. This is the one that seems most common; the Heimskringla (an Icelandic Saga) describes Viking warriors moving from one ship to another.

Projectiles are a key component. Bow strings, however, must be kept dry, as would gunpowder. The Mary Rose had archers, and her remains yielded the only known longbows from this era (C16th). She also had gunports (the earliest known archaeological example of) so she had guns (cannon) to go with them. But bows and small guns would not had had the range of large cannon, so close quarters would still have been preferred for them to be effective. Mary Rose had a forecastle and an aftcastle - this is where the archers would be stationed as to have the best view to fire down onto an enemy ship.

3. Ship as weapon

This means you have the weaponry where much (or all) of the fighting is done without hand-to-hand combat, and may result in defeat/surrender without the two ever coming into physical contact. Rawlings (2007), in the The Ancient Greeks at War describes the use of rams on triremes - the tactic was to try and hole the enemy ship and incapacitate her. Given that triremes were controlled during engagement by three banks or rowers, the strengths (manoeuvrability) is weighed against the weaknesses (lose your oars or lose your rowers: go nowhere). Triremes also operated in formations; it fair to say that the a mass of ships, manoeuvring very close to each other and ramming each other would have resulted in hand to hand combat on the decks of these ships. Triremes would have had archers and marines on board, so this was expected.

Later in the age of sailing ships, ships as weapon means gunpowder and guns. Fighting on deck is part of the engagement, but you get to bash the crap out of each other first to gain an advantage, be it in structural integrity, the ability to run away, or simply killing as many of them as possible before boarding. Ideally, they surrender before you board.

The Greeks (certainly the Athenians) had standard naval tactics; they aren't hard to find online. The Vikings are a bit harder - you have to read a lot of sagas to get the bit you need to find. In medieval Europe there are sources and shipwrecks, although I would have to do more research to dig up more deatils on thier naval tactics.