As late as 1571, in the Battle of Lepanto, both sides relied on ships with oars. But past that point sails-only ships start taking over Mediterranean sea battles. What finally tipped the balance to sails?
Cannon tipped the balance towards sailing warships.
The more powerful and effective cannon became, the further the balance tipped.
Galleys can only mount a few cannon at the bow. Sailing warships can mount many cannon along each broadside. (That space is taken up by oars on a galley.)
The advantage of galleys was that they could move and maneuver when there was no wind. This is why they lasted longest in waters where the winds are fickle, like the Mediterranean and the Baltic.
Besides their lack of cannon, galleys had another disadvantage in that their range was short. The need to feed and water many oarsmen (as well as crew and soldiers) from a long, narrow, and shallow draft hull (to make the galleys easier to row) meant that not enough supplies could be carried in the hold for long range cruising. This also restricted use of galleys to narrow seas with many ports, such as the Mediterranean and Baltic.
In the battle of Lepanto, the Christian fleet had six galleasses, as well as 206 galleys. Galleasses were new design of hybrid ship, developed by Venice (all 6 galleasses in the Christian Fleet were Venetian). They were an attempt to combine some of the hull strength and ability to mount many cannon of sailing warships, with the ability to move under oars of galleys.
In the battle, two galleass were deployed well in advance of each of the three divisions of the Christian fleet. As the Ottoman fleet of 222 galleys and 56 galiots (smaller galleys) advanced towards the Christians, they were bombarded by the heavy cannon of the galleasses, and 70 of them were sunk before the main fleets clashed.
Later in the battle, the slow galleasses had a hard time getting back into the action, but one of them arrived in time to save the Christian Northern Division, and the Christian fleet went on to win.
Despite the galleasses being one of the reasons for the Christian victory at Lepanto, the design never proved very popular. The galleass was not as powerful or mobile as a sailing warship, and nowhere near as fast under oars as a galley.
Still, the galleasses at Lepanto demonstrated the power of massed heavy cannon against galleys, and it was that power that led navies to use sailing warships more and more and galleys less and less.
There are probably a lot of factors, but I'd argue that the most significant was the opening up of the Atlantic. Long after Lepanto oared vessels were still in use in the Mediterranean, but the defeat of the Ottoman fleet there really spelled the end of major Ottoman naval activity. The other nations with naval interests in the Mediterranean (I'm thinking primarily of Spain and France) were beginning to look West by the end of the 16th century and suitability for Atlantic waters became an important factor in shipbuilding. It was not just shipbuilding: galley warfare required different tactics to warfare predominantly involving sailing ships with guns mounted down the sides, so the experience of admirals and captain gradually tipped in favour of sailing ships over galleys as the Atlantic theatre became more important. This is probably a question which could be debated ad nauseam, so that's just my two-ducats' worth.