Consider the deck of a ship of the line. Everywhere you look above you is absolutely lousy with ropes and sails. And, usually, men.

First off, where would you even store a trebuchet on board? Especially one large enough to have a significant range for sea battles. Secondly, how would you secure it to the deck so it didn't fall off or wreck the boat when it fired? Thirdly, and most importantly, how would it fire without getting entangled in the rigging and especially without hitting the rigging with its projectile? And how would you hope to fire the thing with any accuracy? With cannons they go off quickly enough that you can time their fire. Additionally, they are fired in a broadside, which covers for the inaccuracies induced by firing from a rolling ship at sea. With a giant trebuchet it takes several seconds for it to launch its projectile, and you have no broadside.

But that's a sideshow really to the major issue. What insane captain would allow the crew to set anything on fire on the deck, even if it is intended to be flung to another boat? Especially in the chaos of eighteenth century naval warfare, imagine a crew carrying an incendiary load to the basket of the trebuchet and then setting it on fire, all while enemy cannon and musket fire whizzes past. It would be so easy for a vessel firing such rounds to be "hoist by their own petard" as the incendiary material was knocked loose before firing and destroyed the wrong vessel. Especially since a trebuchet on deck would be incredibly vulnerable compared to cannons on a gun deck.

However, something that was more feasible at the time was the use of "hot shot" or "heated shot". This typically required the use of a mobile furnace that could heat a cannonball to red hot before firing. This was extremely dangerous because mishandling could result in catching fire to the ship, and there was also a risk of premature detonation of the gunpowder in the gun during loading. However, hitting a ship with a hot shot could be devastating. Often the loads used less powder so that the balls would lodge in the enemy vessels rather than punch through. French and American Naval vessels during the 18th and 19th centuries sometimes used heated shot, though there are only a handful of cases where it was effective at sea. However, it was generally not a common practice over a long period of time due to the risks involved.

Much more commonly heated shot was used from shore batteries, where the risk of destroying one's own ship was not an issue.