Expanded from an earlier answer:

Like a lot of things about sailing, it depends. I realize that's an unsatisfactory answer, so let's expand on it a bit:

Generally speaking, the fundamental determinant of a ship's speed (assuming we're talking about a displacement hull) is tied to the length of the hull. There's a formula for this: HS = 1.34 x √LWL; or to say it in English, hull speed is equal to 1.34 times the square root of the length of the waterline. In practice, though, most modern sailing hulls are capable of exceeding this -- the ratio is tied to the bow wave and stern wave a hull generates, and as a boat heels or exposes different parts of its hull to the water, those waves and their interactions can change. It's kind of useful to think of it as a minimum maximum, but the overall lesson is that as a hull gets longer, its potential speed gets greater. (Feel free to click here for a more technical discussion.)

That of course doesn't take into account hulls that are planing hulls or foiling hulls, etc., because those didn't exist yet during the time period you're asking about.

So let's talk about what would determine speed of a ship in the Age of Sail. First off, most large warships in northwest Europe and the Mediterranean were square-rigged, that is, they had square sails on their fore- and main-masts, and often square sails atop their mizen (it wasn't unusual for the mizen to have a spanker or something similar that was gaff-rigged; the mizen was used for turning more than driving the ship, and many ships up until the 1770s or so had lateen-rigged mizzens). The ship would also carry fore-and-aft sails extended by its stays (staysails) and by its bowsprit (jibs and kites). This type of rig is most efficient moving downwind, but it would allow a ship to tack up into the wind or wear against the wind, to make good progress upwind (into the wind direction) by sailing on tacks against it.

This is where displacement and draft come in.

Generally speaking, square-rigged ships (like all sailing ships) suffer from leeway; that is, they are affected by the wind by being blown somewhat off course on most points of sailing that aren't directly downwind, or in front of the wind. So the speed of the ship through the water is often different than its speed "over ground" as it were, that is speed relative to the land. All other things being equal, ships with a greater draft will often sail with less leeway than ships with shallow draft, because the wetted surface under the water will prevent some of that leeward motion. (Dutch ships, for example, had to be built with shallow drafts to get in and out of their main ports, but this hampered them in working into the wind and up/down the Channel.)

Something else to consider is the effect of draft on the stiffness of the ship -- in this case, stiffness meaning the amount it would heel in a wind. All ships have some sort of heel, but the stiffer a ship is, that is the more its masts can stay perpendicular to the water, the more of the force of the wind can be transferred to the masts and hull.

But the main determinant of a ship's speed was the amount of inertia that its masts would have to move -- all things being equal, a frigate during the Age of Sail could spread a similar amount of sail to a first-rate, but do that on a much smaller hull, so those ships were generally faster. The sail-to-hull ratio, as it were, is greater.

And particularly small ships such as cutters or Bermuda-rigged ships could be quite fast, in most seas. Those ships might struggle more in heavier seas, though -- all things being equal, a larger hull will retain more momentum when becalmed or when fighting through large seas than a smaller ship.

That's all very general, though -- one thing to keep in mind is that French ships brought into service in the Royal Navy often outsailed similarly built ships sailed by the French, because British crews and officers had much more experience at setting sail and trimming ships to get the best out of them, and were at sea much more often.