How was naval warfare conducted before cannons were placed on ships and boats?

In the Mediterrenean, galleys were the most common ship of war since ancient times. This vessel could use sails, but in battle it was rowed and the standard tactic was to ram the enemy vessel with a large ram in the front of your own vessel. Ideally, you could ram the enemy vessel at 90 degrees, but in practice, this was hard without destroying the ability of the enemy vessel to manouvre.

The galley was rowed by a large amount of slaves (or in some cases, conscripted, corvee or paid labour) and attacking the rowers with javelins, slinged projectiles or archery was common. However, on the larger galleys, the rowers were usually protected under a half (being open in the middle along the full lenght of the ship) or even full deck.

The best way to destroy the manouvrability of an enemy vessel was to destroy their oars. If enough oars where destroyed, the enemy vessel would be dead in the water and an easy target for ramming. All vessels carried spare oars for such occasions, but they took time to deploy.

Galleys grew larger and heavier to be able to deliver more weight into a ramming and to be better protected as the Mediterrenean powers grew larger and richer and were able to afford such vessels. Monoremes became biremes, which became triremes which became quadiremes which became quingiremes and so on. The larger vessels were less manouvrable, but often so heavy that they were immune to ramming from smaller vessels. The Romans started to use boarding as a primary tactic to deal with the better handled and built Carthagianian vessels. The corvus, essentially a heavy plank walkway with a large iron nail at the end was dropped on an enemy vessel, fixing it in place and creating an easy way for Roman marines to board the enemy vessel.

During this era, sailing was rudimentary, and often required the right wind conditions, as ships were unable to cruise against the wind. A military or pirate vessel that wanted to capture a trading vessel (which relied on sailing only) only needed to place itself downwind, wait for a trading vessel to appear sailing on the wind and row against the wind to capture it. The sailing vessel was unable to flee since it could not sail against the wind.

Galley naval warfare continued to be the primary way to fight at sea in the Mediterrenean throughout the medieval and renaissance. The last large-scale battle fought by galleys was the Battle of Lepanto 1571, well after the inclusion of guns aboard naval ships.

With large fleets, formations and fleet tactics evolved as well, with some ships aiming to break the oars of the enemy vessel, or carrying missile troops to attack the rowers, and ships following after doing the actual ramming.

Galleys were small and light (to be fast and manouvrable to be able to ram) and carried a lot of crew (there were massive numbers of rowers aboard) and were thus not very sea-worthy in heavy waters nor able to store provisions to remain at sea for long. It was common with smaller galleys to pull them up on the beach and camp ashore during the night. The need for large amount of water in the hot Mediterrenean climate and the limited amount of storage aboard a galley meant that the ships had to resupply often. Thus most galley warfare happened close to a coast.

The need to stay close to the coast also created siege-like situations, where one fleet could blockade another, attempting to force it to attack under unfavourable conditions (such as in cramped waters where they would have difficulties manouvering) when they ran out of water or other supplies.

Fire ships (ships loaded with flammable materials, set ablaze and set on a course for an enemy fleet in an attempt to catch several of the enemy vessels on fire) were used in such situations, one of the first instances known is during the Athenian's Sicilian expedition, when the Syracusans attempted to use fire ships against the Athenian fleet.

The early medieval era saw the addition of a new weapon for naval warfare - fire. The Eastern Romans (Byzantines) started using siphons (pumps) to spurt Greek fire, a flaming liquid that would not only stick - it would also float on water and keep burning - akin to a modern flamethrower in naval warfare. Using this Greek fire was risky though. If the pump broke or the container was punctured, you risked a huge fireball among your own fleet.

The Arabs used naphta - a flammable liquid poured into glass or thin clay containers with a rag or fuse attached. The fuse was lighted and the container thrown at the enemy vessel, where it broke and created a small sea of flames akin to a modern molotov coctail.

In nothern Europe, the vikings used archery and spear throwing against each other and others on their open longships and then boarded. Longships were rowed in battle, in fact, the mast and sails were usually taken down to be out of the way during battle. Attempting to kill the man handling the rudder with archery or spearthrowing was not unheard of.

In the north Sea and Atlantic, the French and English would use cogs, sailed single-masted vessels with a high tower or castle-like structure fore and aft for men to fire arrows and fight from. The English made extensive use of longbowmen on their vessels and their ability to rain down arrows upon the French vessels. The cogs were usually open or one-decked and there were little protection against missile fire.

So, to summarise - ramming, boarding, fire, fireships, missile fire (slingers, archers, javelin and spear throwers).