Mutual sinkings were rare in any war that I am aware of. One situation that is similar to the topic of your question came during the Battle of the Heligoland Bight, during WWI. The battle was a British attempt to strike German patrols in the Heligoland Bight using light forces (destroyers and cruisers) supported by battlecruisers under Beatty. The early stages of the battle saw British destroyers and cruisers tangling with a smaller force of German destroyers. During this engagement, the German torpedo boat (small destroyer) V-187 was hit heavily, and began to sink, assisted by scuttling charges laid by her crew. A flotilla of British destroyers approached the stricken ship, and Defender lowered her boats to take off the crew. However, just eight minutes after this had begun, the appearance German light cruiser Stettin forced the British ships to withdraw. Defender' was unable to recover two of her boats, abandoning their crews to what seemed like certain captivity. Stettin declined to pick up the boats, first steering to engage the British destroyers, and then withdrawing to repair damage sustained in the engagement with them. The crew of Defender's boats took this abandonment in their stride, bringing a considerable number of German survivors aboard. A total of 28 Germans (18 wounded, ten unwounded) were taken aboard the boats, which were crewed by just ten Britons between the two. The German survivors were well treated, with the wounded being looked after well. Fortunately for both the British and Germans alike, they were being observed by the British submarine E-4, commanded by Lieutenant Ernest Leir. Once the coast was clear, Leir took E-4 to the surface, and approached the boats. Leir took the British seamen aboard the submarine, along with three injured Germans. He left the rest of the wounded in the boats, along with the unwounded men, giving them food, water, and a compass with which they could steer their way back to German-held territory. This incident was typical of behaviour between and towards men in the water. Sailors always saw the sea as being as big an enemy as whoever it was they were fighting. A survivor could no longer hurt you; instead, the impulse was to help them - at least in part because all involved recognised that it would not take much for them to be in the same situation.

Follow up question; using your example of trafalgar, would an English rescue boat have been sent only to recover English sailors or French also?

It can be a bit difficult to generalize about things like this, but generally speaking during the period I study, men would make every effort to rescue one another, and succor the enemy wounded, once a ship had surrendered. There were some exceptions to this rule that were situational -- for example, European sailors did not tend to show much mercy to corsairs in the Mediterranean (and vice versa), but for the most part the customs of the sea was to whenever possible help shipwrecked or distressed mariners, even if you had been fighting moments before.

The usual way that a ship in action during this period would surrender would be to strike (haul down) its flag, which would indicate that the men aboard had stopped fighting and would offer no more resistance. The custom of striking the flag and then respecting a truce was so common that it occasionally led to confusion and friendly fire -- when USS Chesapeake struck to HMS Shannon, Shannon's lieutenant George Watt attempted to hoist the White Ensign over the American flag, but tangled the halliards so that the American flag rose first, as if in defiance. Shannon saw the flag raising and fired on Chesapeake again, killing Watt.

After the battle, the surviving American soldiers were held in Chesapeake's hull, with cannon loaded with grapeshot pointed down the hatches to keep them from attempting to retake the ship. The American wounded (including Chesapeake's captain, James Lawrence) were taken aboard Shannon for treatment; Lawrence died of his wound.

But that's not quite the question you were asking -- in the case of a ship that sank during the battle, what would happen to those survivors? u/thefourthmaninaboat pointed out that only one ship was sunk during Trafalgar (though more were lost in the storm that followed the battle) and it blew up. If a ship were on fire, and the fire were clearly out of control, men would attempt to escape by any means necessary and other ships' boats would often try to rescue them, which took a deal of courage. (When HMS Queen Charlotte caught fire and burned off Leghorn in 1800, several American ships tried to render aid but were deterred by the ship's guns going off at random as the heat of the fire grew. Queen Charlotte blew up, taking 673 men with her.) In the case of other actions where a ship sank during a battle, l'Orient blew up during the Battle of the Nile in 1798, but that happened at night -- survivors were rescued in the morning. At Quiberon Bay in 1759, Thésée was sunk by gunfire, Superbe capsized and Héros struck her flag then ran aground. That battle was unusual in taking place during a gale, and so men on ships driven ashore were rescued from the shore.

Once men were captured in battle, they would be kept as prisoners (the Chesapeake survivors were kept at Dartmoor Prison in Britain.) The French Revolutionary and Napoleonic wars were the first in which there were not mutual exchanges of prisoners as there had been in earlier wars -- in 1794, the French Revolutionary government actually called for the execution of any British or Hanoverian prisoners, though only one captain seems to have carried out those orders. The French Directorate briefly resumed prisoner exchanges, but Napoleon reversed that policy and also refused to pay for the support of French prisoners in Britain, which led to a bit of a manning crisis in the French navy by the end of the wars. Something more than 80,000 seamen were kept prisoner by the British government by 1814, the seamen first in hulks then dispersed inland, and the officers living on parole.