We all know of the venerable line of battle, used throughout 1650-1850, with large ships of the line pouring broadsides into eachother till one side struck their colors. Likewise the galley fights of Roman times till the 15th century, focused on ramming and boarding, are something we can all visualize.
What about that middle period, from 1500-1650? Caravels, carrack, and early galleons all were dominated by huge fo’c’sle and aftercastle architecture, and mixes of forward facing chasers and broadside Guns, but were also ridiculously top heavy.
How did these ships fight, both in single ship actions as well as in fleet battles?
This is a good question, and I've written sailing tactics before a few different times. I'll link some answers below. The short answer is that it took awhile to work out how to use guns (and even where to mount them for best effect), and that a lot of naval battles sort of fell apart into 1:1 melees where ships would try to maneuver to fire a shot or two before closing and boarding -- there wasn't a goal of two fleets just hammering away with quickly reloaded guns.
We see this particularly by the time of the Spanish Armada, where the English fleet sort of peels off one ship at a time, "charges" the Armada sailing in a group, and fires off all the cannons before retiring to reload. I mentioned this in this previous comment, but during this time ships' cannons were not usually reloaded inboard, but over the side of the ship. The English ships seemed to have averaged about 1 to 1.5 shots per cannon per hour; the Spanish ships (which to be fair were under frequent harassing fire) about one shot per gun per day. The goal of the Spanish fleet, of course, wasn't necessarily to engage the English fleet, but to link up with the Duke of Parma's army in the low countries and to transport that army to England.
Interestingly, the line of battle very seldom produced decisive results in battle, and one of the tactical innovations in the Napoleonic wars was to use tactics that would break through an enemy's line of battle and produce melee-type battles where ships that had better trained sailors and who could fire faster would be at an advantage.
Some other links that may be of interest of you:
https://www.reddit.com/r/AskHistorians/comments/3ebdk5/how_did_nelsons_tactics_work_at_trafalgar/
http://www.reddit.com/r/AskHistorians/comments/2fs8qr/how_much_more_advanced_was_a_british/
http://www.reddit.com/r/AskHistorians/comments/29f3s7/how_does_the_royal_navys_organisation_command/
http://www.reddit.com/r/AskHistorians/comments/20x9kz/classes_of_vessels_during_the_age_of_sail/