I know how ship battles worked in earlier times, but I've never really had a clear picture of this.
Royal Navy destroyers fought in pretty much all combat environments that warships fought in during the war. They fought surface ships, aircraft and submarines, and fought in all weathers from Mediterranean sunshine to Arctic storms. As such, it is hard to produce a generalised experience.
For surface actions, yes, destroyers typically fought within visible range of the ships they were engaging. The 4.7in/45 that armed the typical British destroyer of the war had a maximum range of between 15,800-17000 yards depending on mounting. This was just inside visible range from the ship. However, ships would typically engage at much closer ranges than the maximum range. The British torpedoes had a range of up to 15,000 yards, but were typically fired at a range of a few thousand yards.
One of the first major destroyer actions of the war were the two Battles of Narvik, during the German invasion of Norway in April 1940. Narvik and the surrounding fjord had been seized on the 9th April by a German force, transported by ten destroyers. The destroyers stayed in the fjord to refuel from a tanker before returning to Germany. One destroyer was detailed to patrol the entrance to the fjord, while the remainder were moored in the fjord, and in Narvik harbour, along with a number of merchants. Early in the morning of the 10th April, five British H-class destroyers, under the command of Captain Warburton-Lee entered the fjord. Using poor weather and darkness, they slipped past the guard-ship, avoided the ships moored in the fjord, and moved into position just outside Narvik harbour. The leading ship, Hardy, fired torpedoes into the harbour, sinking two German destroyers. Two more destroyers, Hunter and Havock, entered the harbour, heavily damaging 3 more destroyers. Between them, the British destroyers also sank 11 merchant ships before withdrawing. As they did so, they were surprised by the ships that had been moored outside the harbour, which trapped them in a pincer. Hardy was hit hard, including a hit to her bridge that killed Warburton-Lee, lost her steering, and was run aground. Hunter similarly lost her steering, and was rammed accidentally by Hotspur, and sank. However, the Germans had focused fire on the forward three ships in the British line, leaving the remaining two ships to engage two of the German ships at point-blank range, doing heavy damage to the German ships. Following the end of this engagement, both sides withdrew to repair. As the British ships left the fjord, they encountered and sank the German ammunition ship Rauenfels, carrying ammunition to the German squadron, leaving the German ships essentially unarmed. On the 13th April, another British squadron entered the fjord to finish them off. This consisted of nine destroyers supported by the battleship Warspite. A short, sharp, close range engagement followed, in which the British wiped out the remainder of the German squadron in return for only damage to three destroyers.
In a later battle of the Norwegian campaign, the German battleships Scharnhorst and Gneisenau discovered the British carrier Glorious, evacuating RAF aircraft from Norway. Glorious was escorted only by two destroyers, Acasta and Ardent, while the Home Fleet covering force had been drawn off by poor information as to where the German ships were. Heavily outgunned, the British destroyers laid a smoke screen to cover Glorious, then closed to make a torpedo attack against the German ships. Glorious was quickly sunk, then the Germans turned their attention to the destroyers as they made their torpedo attack. Ardent attacked first, scoring no hits, and was then sunk with only two survivors. Acasta's contribution can be described by her one survivor, a Leading Seaman Carter:
The captain had this message passed to all positions: ‘You may think we are running away from the enemy; we are not; our chummy ship [Ardent] has sunk, Glorious is sinking; the least we can do is make a show’. We then altered course into our own smoke screen. When we came out of it I had my first glimpse of the enemy. I fired two torpedoes, the foremost tubes fired theirs. From one of the ships a yellow flash and a great column of smoke and water shot up. We knew we had hit.
After we had fired our torpedoes we went back into our smoke screen. ‘Stand by to fire remaining torpedoes.’ This time, as soon as we poked our nose out the enemy let us have it. A shell hit the engine-room. I was blown to the after end of the tubes. The ship stopped with a list to port. I fired the remaining torpedoes. The Acasta’s guns were firing the whole time. The enemy hit us several times and one big explosion took place right aft. The captain gave orders to abandon ship, and when I was in the water I saw him leaning over the bridge, take a cigarette from a case and light it. He waved, ‘Goodbye and good luck’–the end of a gallant man.
She had torpedoed Scharnhorst, causing heavy flooding and major damage.
During the Battle of Cape Matapan, the British destroyers were deployed to mop up the remaining Italian force after the battleships had crippled a number of Italian cruisers and destroyers. Havock crippled the Italian destroyer Carducci with a torpedo, before sinking her with gunfire. The Australian Stuart sank the Italian Alfieri, again with gunfire. However, one of the odder moments of the battle was to come when they encountered the Italian heavy cruiser Pola, crippled by an aircraft-dropped torpedo early in the battle. Pola was drifting helplessly, and was essentially unarmed. The British destroyers crowded round her, and Captain Mack, of HMS Jervis, decided to board her, as recounted by Jervis' first lieutenant:
After circling round, Captain Mack said ‘I am going alongside—tell the first lieutenant to prepare wires and fenders starboard side’. ‘A’ gun’s crew armed themselves with cutlasses and prepared to capture by boarding. A perfect approach, and over went a heaving line accompanied by the cry ‘Take this, you b-gg-rs’. Uttering blood-curdling cries, ‘A’ gun’s crew swarmed on board. They came back with a story of chaos. The officers’ cabins had been looted and empty Chianti bottles lay everywhere. Only 256 of a ship’s company of 1,000 remained, and they were huddled on the forecastle; the remainder had already jumped overboard. They filed on board in an orderly fashion followed by the commander and captain. A number showed unmistakable signs of inebriation. After about 20 minutes I was able to report that everybody was on board. Casting off, the Jervis steamed slowly round and illuminated the dead cruiser with her searchlight. Captain Mack then ordered the Nubian to finish her off with a torpedo, after toying with the idea of towing her back to Alexandria, some 500 miles. He reluctantly dismissed it owing to the certainty of heavy air attack next day.
Pola would ultimately be sunk by two torpedoes from Nubian.
On the 26th December 1943, a British force engaged the Scharnhorst off North Cape in Norway. A force of cruisers, including HMS Belfast (now a museum ship), forced her southwards, into the teeth of another force, including the battleship Duke of York and eight destroyers. The battle was mainly a fight between the two big ships, but at 18:40, four of the destroyers made a torpedo attack on Scharnhorst. They had closed to within five miles, with two firing starshell to distract from the other two, which closed to within 3,000 yards before launching. Four hits were scored in this attack. The remaining four destroyers closed to finish her off after she was reduced to a hulk. Tristan Jones described the attack:
Our stern went down, the bow shot forward, then we went ... like staghounds at last, at long bloody last, off the leash . ... Faster and faster, plunging and swaying, pulsing with strain until her whole hull screamed in protest . ... All we could see of the nine other destroyers were their pale wakes, until Captain (D) in Onslow opened up with his four-inchers. Crash after crash shook our hull .. . and how the Torps got the tinfish away in any semblance of an aim I'll never know. Out slid the tinfish, long and sinister, and splashed into the darkness
All in all, surface combat in destroyers was an exciting, complex affair. Surviving it was less so. The Royal Navy had neglected its life-saving equipment before, and during the war. The main piece of life-saving equipment was the Carley float or raft, first introduced in 1915. This was an oval life-raft, capable of carrying ten men inside it and ten men hanging on ropes outside it. It also included lockers for food, water, a first aid kit, and survival equipment. Somewhat more rudimentary was the flotanet, essentially a cargo net equipped with floats. This was intended as a way for men to survive until they could reach a Carley raft or boat. Anyone relying on one would stay entirely in the water. Finally, men were, from 1942, equipped with inflatable lifebelts. These had two major disadvantages. It couldn't be inflated in the water because it didn't have a non-return valve, but if inflated, would make a jump into the water from more than 20 feet dangerous. It also failed to keep the head out of the water, meaning that tired or unconscious men would drown. All of these survival methods meant that men would end up immersed in the water. If the water temperature was cold, this quickly led to deaths from hypothermia if the survivors were not quickly rescued. In 1942, Wild Swan was sunk, with the entire crew successfully making it off the ship. However, rescue took 13 hours. Thirty one men died of the cold, and three more drowned following exhaustion. Had the crew been kept out of the water, all would have survived. Convoys were typically accompanied by rescue ships which would quickly evacuate men from the water, but this didn't help men in outlying escorts, or in destroyers with the battlefleet.