The galleys shouldn't be compared on a "more modern or less modern" axis to other contemporary sailing ships, at least not in 16th century which happens to be one of the peaks of 'medieval' galley warfare in terms of size and intensity (discounting the Greeks and Roman times).

Instead we should simple look at the features of each type of ships. Galleys main 'advantage' lies in oared propulsion, making it be able to move interdependently of the wind and giving it maneuverability and easier turning and moving. Making it quite useful in combat, especially if the rowers were free men and potential combatants. The ships were also relatively small, being able to navigate shallow waters, and even directly beach themselves. All in all, a very effective weapon for all sorts of medieval naval and amphibious actions.

The downside was that the ship was low, vulnerable from above and had very limited carrying capacity and due to this operational range. On the other hand sailing ships weren't really that maneuverable. They were tall and large, giving them some protection, but prior to gunpowder weapons, making them very poor offensive weapons compared to the galley. Galleys could frequently surround and attack a sail ship, and overwhelm the defending crews.

With the gunpowder weapons this begins to change, but as always things change with a slower pace over longer time, and 16th century is that time of change. It is in fact the galleys that were the first to be more effective in utilizing the new artillery weapons, by placing ordnance in their bows and with their maneuverability being able to target enemy ships more effectively, and being able to fit larger and longer pieces that could fit in a sail ship. However, the improvements would soon come to their theoretical maximum potential: there was only so much artillery that could fit in the bow of a galleys and only so much weight you can add to be able to row it with sufficient speed. And there was basically not much that could be done to solve this.

On the other hand, over the course of the sixteenth century sailings ships were being built increasingly larger, sturdier, heavier and could fit more and more guns. It took some time to reach it, and it is only by the end of sixteenth century that the familiar broadside-artillery finally sail ship matured in an effective fighting force. Being tall and having armament superiority made it be able to reign down fire on the exposed open decks of galleys, while itself being sturdy enough to survive the hits from the largest galley cannons. Improvements in sail plans were reducing the maneuverability constrains (but which still remained a large drawback) and the large cargo capacity of the sailship allowed it to remain 'deployed' much longer then a galley, another very important asset. The galley remained deployed by some Mediterranean states throughout the 17th century, and even in 18th although with much reduced importance, much like the sea they were operating in.