Oars do have a distinct advantage over sails when there is no wind, however, days with absolutely no wind are very very rare in the area where the major naval powers rose (the Med and western Europe). Added to this, oars have several major disadvantages.
Oars and rowers take up a lot of space. A Galley is essentially a ship where the entire area below decks is used for the rowers. This leaves little or no room for something very important in naval warfare during the 16th century. Cannons.
Oars need a lot of men which means they're very expensive to field and maintain. It also means that they consume huge amounts of food and water, essentially, such a ship relies on operating only where it can be supplied from shore.
In essence, a galley is inferior to a man-of-war in ever case but dead calm.
Others can probably go into far greater depth as I don't have any sources to hand, but I'll just make a couple of points.
The Galley was still used in the Mediterranean (and the Baltic) till at least the early 19th century. The trouble is that galleys were not designed for oceanic sailing and so were of a lighter construction and less weatherly. They also had the downside of needing a large crew to maneuver them, as opposed to a man of war where the majority of the crew could be given over to fighting the guns.
They were successful at what they designed to do (I think there are examples of galleys forcing the surrender of Men of War in little wind and inshore waters) but larger ships were more efficient and adaptable to everything that the sea can, and will throw at a ship as well as being able to project a nation's power far further than a galley can.