I'm familiar with triremes in the classical era and that sailing ships would sometimes use oars to move in dead wind.
But what is this battle (between Denmark, Russia, and Sweden). I've never seen or heard of this, were naval battles with oars to maneuver common in the age of sail?
I hope it's not too late to answer. I missed this post when you asked the question two months ago and have only now found it via the search.
Oar-based naval battles in 18th century were not as widespread and common as during previous era, but they still happened, most noticably in areas where sailing ships often had troubles with maneuvering and simply fitting into the battlefield, i. e. shallow coastal waters and archipelagos consisting of multiple small islands.
Due to ever perfecting nature of naval combat, oar-based vessels generally fell out of favor with European naval powers. Galleys were heavily dependant on boarding actions, and closing the gap became more and more difficult as time went on and artillery systems were getting better and eventually allowed sailing ships to reliably keep boarders at bay. While oar-based ships could pose significant threat to big sailing vessels in 16th and sometimes even 17th centuries, they significantly lacked in firepower and speed and in 18th century it became near impossible to initiate frontal attack without being shredded to pieces or being outrun by sail-based frigates and man-o-wars.
They were, though, still useful in aforementioned areas, where even rare outposts with small garrisons could be very important in controlling large swathes of territory or watching enemy forces. Galleys were dirt cheap in comparison to fullblown ships of the line, were very easy to build and maintain and didn't need as many trained sailors and navigators as sail-based vessels due to majority of the crew consisting of boarding teams. They couldn't operate in the open sea and were largelly helpless against combat ready battleship formations, but were very maneuverable and could be deployed even in very shallow waters or during the calm. I'm unfamiliar with galleys used by western powers during this time (some of them had long and successful galley combat history - maybe Spain continued to utilize them later?), but Eastern European fleets made great deal out of galleys - such as Russia, Turkey and Sweden. The battle depicted on the painting is a battle of Grengam (1720) between Russia and Sweden during the Great Northern War. It's one of the few battles where oar-based vessels managed to score a victory against sailing ships by evading direct artillery duel, tricking several Swedish frigates to run aground and later wrestling control over them in close combat. Other noticable clashes with active use of galleys include battle of Gangut (1714), first and second battles of Rochensalm/Svensksund (1789 and 1790) and naval battle of Ochakov (1788).
Sources:
Sidorov D. A. Swedish Ladoga flotilla 1701-1703. Russia, 2016.
Von Rosen, С. Bidrag till kännedom om de händelser, som närmast föregingo svenska stormaktsväldets fall 1 Sveriges ställning till Polen 1701–1704 / Carl von Rosen. — Stockholm: Norstedt, 1936.