The overly glib answer to this is that they weren't invented yet, or at least not invented in a manner that would be useful. Although explosive shells date back to the 1300s, around the time of the invention of cannon, shells of the 18th c. were fused by an actual fuse (think of a cartoon bomb with a length of gunpowder) rather than a contact or some other type of automatic fuse. Although such a fuse could be ignited by the hot propellant gases when a shell was fired, it was more often cut to measure and ignited right before the shell was fired. The idea was that the shell would explode at the proper time, which was a tricky business to get just right.

Now there were ships of this period that did employ shells to, uh, shell enemy positions, mostly by firing the shell at a high trajectory with the idea of dropping it behind fortification walls or into entrenchments. So-called "bomb ships" would have one or two howitzers fitted right forward, and would be fired by anchoring the ship at double anchor and aiming the gun or guns by means of a spring line (this gets to the idea) attached to the anchor rope. Cutting the fuse just right so the bombs burst in the fortress, rather than bursting in air, was skilled business.

As far as firing shells horizontally, there were experiments on doing this from land-based howitzers starting around the time of the American Revolution, but they never took hold in the Navy. (The notable shell of the Napoleonic period was the shrapnel shell, adopted with great enthusiasm by the British army after 1803.)

It's hard to say exactly why something didn't happen, rather than did, but I would say that my opinion is that the fact that ships had a system of laying down gunfire that worked well and informed their doctrine; and that putting explosive shells on a wooden ship increased the number of things that could go badly wrong in battle; probably combined to delay their adoption until well after the Napoleonic period.