The 18th Century was towards the latter half of the Age of Sail, and so by this time sailing ship tactics were well defined and pretty much universal across all navies. In order to fully utilize the strengths of a ship’s broadside and to avoid hitting friendly ships, navies used a tactic known as the line of battle. In a line of battle a fleet of ships would line up end to end and would sail parallel to the enemy fleet, who were also in the same line. The two lines would sail next to each other and would exchange fire until one fleet was destroyed or retreated. Naval engagements were long affairs, with fleets often battling for hours and sometimes for more than one day. The Four Days Battle is famously one of the longest naval engagements in history, and as the very creative name suggests, took place over four days. This battle was in the 17th century so its outside the timeline of the question, but it still shows that naval battles could last for quite a while.

In most battles the side that held the weather gage—a term meaning that the fleet was upwind of its enemy—held the advantage. The fleet that held the weather gage was able to decide when to engage as they could maneuver at will towards any downwind point while the downwind fleet is incapable of sailing towards the upwind fleet. The advantage to being downwind was that you could quickly turn to retreat.

By the time of the 18th Century naval tactics were beginning to stagnate, and the traditional line of battle was becoming less and less effective. The Battle of the Chesapeake was a strategic French victory, but in two hours of fighting the French only damaged 6 British ships, and only one was damaged enough to be scuttled. Some controversial new innovations took place during the American Revolution—such as the breaking of the French line of battle by the British at the Battle of the Saintes, and the new tactic of focusing fire on the rear of an enemy’s line—but the line of battle would remain fairly consistent throughout the 18th Century.

One on one ship battles, while not common in large battles when utilizing the line of battle, did still occur. The tactics in a duel were still fairly similar to tactics in a large battle. The ship holding the weather gage could engage whenever it wanted and held the advantage. An Essay on Naval Tactics by John Clerk suggests that the ship holding the weather gage should sail toward the stern and the wake of the opposing ship. By keeping close to the back of the ship, the engaging ship could easily fire into the vulnerable stern while also avoiding the majority of the enemy ship’s broadside. The Essay also states that when in a one on one battle, and if trying to avoid a close melee, it is best to fire into the rigging as that cripples the ship and allows for an easy retreat. One on one battles were a little like the deadly waltz you mentioned in your question as each ship tried to gain an advantage on the other; the two ships maneuvered to get a better placement for firing off a broadside into the vulnerable parts of the enemy while also trying to avoid getting hit.

The Golden Age of Piracy ended within the 18th Century, but I still think it counts so I hope you don’t mind if I mention pirate tactics. Since pirates wanted to capture and plunder a ship they generally tried to avoid firing their cannons. Pirates would attempt to use intimidation to frighten their targets into surrendering, and if that didn’t work they would sail close and use grappling hooks to bring the ships together for boarding. If the target ship fired at the pirate ship, then the pirates would resort to firing at the rigging or at the crew in an attempt to immobilize the ship. Firing on the ship was only a last resort however.

As a sort of fun fact, it is said in An Essay on Naval Tactics that the French crews were trained to fire into the rigging, while the British preferred to fire into the hull of the ship. Both schools of thought had their advantages, but in a larger battle firing into the hull could be far more devastating. One of the reasons for the great success of the British in the Battle of Trafalgar is supposedly because they obliterated the French crews while the French were only firing into the British sails.

An Essay on Naval Tactics by John Clerk

The Influence of Sea Power upon History by Alfred Thayer Mahan

Fighting Instructions, 1530-1816 by Julian Stafford Corbett