Let's say third rates, roughly 80 guns. What I am referring to is an exchange of round-shot fire between the two ships' gun decks, at medium range, directly into each others' side hulls. I am NOT referring to raking fire from aft to forward or vice-versa, nor am I referring to shot into the rigging, the crew, etc. I am interested to know how much actual damage would be done by such an exchange, and how many of the cannonballs would simply bounce off harmlessly, or nearly so. Do we have a historical example(s) to look to for evidence of such an exchange, despite it being against most doctrine?
Again, imagine medium range, so no carronades, mortars or the like, and imagine the two ships are essentially equivalent in armament, construction, and design.
There's no good answer other than "it depends," unfortunately, so I'm not going to answer this directly -- it would just be speculation. Instead, I'll break this down into factors to consider, then talk about the historical evidence we have.
Even with ships of roughly the same rate, there would be many factors to consider:
how new are the ships? How seasoned is the timber on each ship? How thick are the scantlings and other support members, and what is the main means of construction (treenails in the British/Dutch navies, nails in the French, mixed in Spanish)
What kind of wood are the hulls made out of? Oak was the traditional structural material in Europe, but British ships were made of teak in the Indies and the Spanish had yards building out of tropical hardwoods in the Americas, most famously in Havana.
What weight of gunfire is each side throwing? The French pound was not the same as the British pound, so guns of the same "weight" wouldn't necessarily throw the same actual weight of ball.
Where are the guns aimed? It was a bit unusual for British ships, for example, to suffer wounds in the hull because French doctrine was to fire high to disable rigging.
How are they charged? Some British captains liked to double-shot their guns, at least for the first broadside.
Are the guns hot? I consistently read that artillery of the Napoleonic period fired farther if the cannons had warmed up from firing (I'm not sure of the technical details, but I would assume that the heated metal reduced windage inside the barrel somewhat).
With all those factors in mind, then, let's look at some of the types of damage that a wooden ship would take. Keep in mind that guns of the time fired solid shot (in the scenario that you're describing) at fairly low muzzle velocities (after the Spanish Armada, for example, the English navy found that many of its shot simply went through and through the Spanish ships without having a "smashing" effect, so they reduced powder charges and experimented with different calibers of guns).
A cannonball hitting a wood hull at fairly low velocity is consistently described as having a "smashing" effect. The most obvious effect of this would be to make a hole in a ship's hull, the size of the ball itself or larger. Several balls hitting the same section of a ship at once, or soon after one another, could easily beat two gun ports into one, or cause other problems (flooding if the ship were hit below the waterline, which could happen as the ship rolled in the seas).
The smashing effect of balls would lead to what we'd later refer to as a "spalling" effect inside the hull. Splinters are consistently mentioned in primary sources as being a major problem for the men aboard ships (big pieces of oak hit at a low velocity with sharp edges caused appalling wounds). (The Mythbusters TV show attempted to "bust" this myth a few years ago, but as I recall they used pine wood, not seasoned oak, and a high-velocity ball, which does not reliably replicate a naval battle, and in any case sources are full of mentions of splinters.)
Balls entering the hull could cause major damage to the stuff inside the hull; when a ship was in a gun battle, its gun decks would be stripped of everything except guns and men (screens, partitions, furniture, etc. would be removed). So a ball entering the hull would spend its force on either a gun, a gun carriage, a man, or some tackle or equipment that was still there for firing the gun. We hear of damage occurring when guns are knocked over or off their carriages, or when their breeching-ropes would be cut and the gun would run loose (a large naval gun could build up plenty enough momentum to plunge over the side of a ship). The upsetting of guns would cause damage on their own as they rolled around inside the ship, until tamed by the crew.
Balls entering the hull could also easily knock over match-tubs or scatter other flammable materials about, which could cause fires.
The inside of the hull, of course, also housed the lower part of the masts of a ship (they pierced through the gun decks to sit on the keel), which could also be damaged in battle.
And, of course, a cannonball hitting a man directly would cause major damage.
I hope this helps -- and that you understand why it's hard to answer definitively.