I can only speak to this from the northern European perspective, but what we'd today call "command and control" were huge problems for navies of this time. So let me break this down for you.

I understand most fleets had a flag system to communicte

Actually, in the 17th century (the 1600s) this wasn't necessarily true. Admirals made up ad hoc signals to control fleets in battle, and ordered their crews to follow certain signals, but a great deal of battle was improvisation. At least in the British navy, there was no widely adopted book of flag signals until 1790, and signals were not standardized until 1799. So that's about 100 years after your time period.

but I doubt that could be of much use in a huge naval battle where there where dozens of ship on either side.

Well ... it depends, of course, on the use of the flags and how easy or difficult it was to see them. The naval battles of the 17th century were very much set-piece battles where the line of battle was holy and a captain might be punished from deviating from it. That was partly to deal with the C&C issue, and partly to make sure that whatever signals could be sent would be seen by all. (Eventually, the admiral's flagship would be in the middle of a line formation and frigates outside the line would be repeating signals to other ships -- this worked until powder smoke and close action would make it impossible to see.) Flag signals were generally used before and after battle; exact tactical command in battle was difficult or impossible, even when navies had wireless.

Also; Was each captain responsible for his own stategy or was there a predetermined plan they had to follow? In other words, how exactly did large naval battles work in the 17th century?

In the 17th century, fleets generally formed into a line of battle to attack the other fleet. That is, ships lined up bow-to-stern in a following line, which maximized their broadside fire. Now, ships at the time were very similar to one another, broadly speaking, so only a major tactical blunder could give an admiral a significant advantage over another, so a lot of sea-fights degenerated into a rather inconclusive meeting of the lines. The eventual solution to this was to turn perpendicular to the enemy's line and break through it, as we see at the Battle of the Saintes, Trafalgar, etc.

If you're really interested in command and control at sea, I'd recommend Andrew Gordon's The Rules of the Game: Jutland and British Naval Command.