The main problems are accuracy/velocity and damage.

We'll start with the easy one first: damage. Your average HEAT round makes a very small hole, and the jet of molten metal it shoots is roughly six-to-ten feet long. While lethal to something the size of a tank, where you have ammunition, fuel, engines, and crew all huddled inside a compartment protected by a few inches of armor, even a small warship such as a destroyer is much, much larger. As tank guns are roughly the size of a 5" naval gun already, increasing the size of the shell isn't going to cause that much more damage from a HEAT round. In effect, you'll be drilling small holes into the surface of the ship, but unless you got really lucky with a destroyer and hit something vital, you're not going to really be hurting it much. When you start to move up to battleship-level armor, WWII-era HEAT rounds may not even be able to punch all the way through. Meanwhile, the larger you go with a standard AP round, the more armor you can punch through, and the more damage you'll cause when the bursting charge goes off.

The more difficult reason to explain is the one of accuracy. In short, HEAT rounds are designed to be fired from low-velocity guns; indeed, that's why they were created in the first place, to give big, slow rounds the ability to punch through armor. While HEAT is fine at shorter ranges, spin-stabilized high-velocity rounds are simply more accurate at longer ranges.

The problem here for HEAT rounds is that at sea, everything is at long range. Battleship guns can reach out and touch you from 25 miles away. The US Navy's 8"/55cal gun carried on their heavy cruisers had ranges of about 15 miles. Your average HEAT round from a tank begins to dramatically lose accuracy at one kilometer or so.

Then, finally, most bigger ships have spaced armor. Oh, it's not called that, certainly, but that's the effect. A problem with HEAT rounds is that the jet will only travel a certain distance. When triggered, it begins to burn at that point, which is why you see tanks carrying what looks like skirts or fencing on their sides: to prematurely trigger HEAT rounds so the jet burns through air before hitting the important stuff, like the hull of the tank. A ship's deck armor might have a 2" thick layer designed to trigger incoming AP shells, then have a thicker layer of armor a level below that, to prevent the blast from the exploding shell from hurting anything. It would do the same to a HEAT jet.

You're right, though: a smaller weapon did need to be developed to allow a smaller ship to fight a larger one, and one was: the torpedo.

The modern shaped charge wasn't invented until the years immediately before World War II.

The hollow cavity effect was demonstrated as early as 1883, but Franz Rudolf Thomanek of Germany and Henry Hans Mohaupt of Switzerland were the first to create a practical shaped-charge explosive. Thomanek's hollow charges were first put into use at Eben Emael, Belgium on May 10, 1940, while Mohaupt has a more interesting story.

He demonstrated his cavity charges at the Swiss Army Proving Ground, his laboratory and at the French naval artillery proving ground. The French and British were interested, but Mohaupt was asking a lot of money, and they didn't want to pay it. Instead, they realized what principal Mohaupt was using and figured that if he could figure out how to make it practical, so could they. In November 1940, British soldiers were issued the No. 68 grenade, the world's first hollow-charge antitank rifle grenade.

Mohaupt, meanwhile, was attempting to sell his design to the U.S. Unfortunately for him, the U.S. also realized what he was doing, and signed a contract with him at a fairly low price. They then classified the design -- excluding him from its development, because he was a foreign national -- and turned it into the bazooka.

If you're interested in reading more, check out William Walters' paper, "A Brief History of Chaped Charges," presented at the 24th International Symposium on Ballistics in 2008.