What sails a ship could set varied by period. I will answer for a full rigged ship in the early 1800s, Napoleonic War period.

The ship would have four square sails on each mast, lowest to highest; course, tops’l, t’gallant, royal. (except the mizzen mast, which usually would not carry a course, but a fore and aft gaff mizzen or spanker instead)

Besides the spanker, the ship would carry fore and aft stays’ls. Generally 3 between the fore and main masts (main stays’l, main top stays’l, main t’gallant stays’l) three between the main and mizzen masts (mizzen stays’l, etc.) and one forward of the foremast (fore stays’l). On stays between the foremast and the bowsprit and jib boom, forward of the fore stays’l, there would usually be three jibs (inner jib, outer jib, flying jib). Additionally, stuns’ls could be set which extended the square sails laterally.

Aeson Daandryk is incorrect when he says that fore and aft sails were used when the wind was abeam, and square sails used when the wind was abaft. Square sails could be braced around to power the ship when the wind was coming from well forward of the beam. Square sails and fore and aft sails were almost always both set in most circumstances, except in very heavy weather, when only one or two lower stays’ls might be set.

The main purpose of this array of sails was to cater for varying wind speeds.

Sailing ships encountered three main categories of wind speed:

Very light winds, from no wind, through barely enough wind to move the ship, to not enough wind to move the ship near full speed. In these conditions, all sail would be set. The very light winds would often be slightly stronger higher up, where the friction between wind and wave did not slow them as much, so high sails were valuable.

Ordinary winds, all the way from full breeze to gale. In these winds a ship could theoretically sail near maximum speed (hull speed in knots is circa 1.3 times the square root of the water line length in feet). When the winds were lighter, all sail would be set. As the wind grew heavier, sail would be taken. The ship could sail at maximum speed with less sail area in stronger winds, so there was no point in carrying more sail (in fact, it would slow the ship down, due to heeling more), and there was risk of damage in carrying the lighter sails as the wind grew stronger.

Very heavy winds, strong gales, storms. In these conditions, sailing near maximum speed became dangerous. The risks of sailing her under a giant wave, of broaching down a wave, became too great, and the risk of damage to sails and masts from carrying sail increased. Here sail was reduced even further, until only storm sails were set, or possibly she would run under bare poles with no sail set at all.

So, in light winds, all sail was set. As the wind grew stronger sails were taken (individual sails could also be reefed to reduce their sail area). When taking sail, the higher sails were taken first. There was one exception to this. Courses were generally taken before tops’ls. This was because: Courses were more difficult to trim to the wind than tops’ls, as to trim courses, tacks, sheets, and braces all need to be adjusted, whereas to trim tops’ls, only the braces need to be manned; because courses are vulnerable to being hit by large waves in high seas, which risks damage; and because tops’ls are easier to reef than courses (because the yard can be lowered by the halyard, whereas on a course, the yard is fixed in height, and the sail needs to be fisted up to the yard for reefing).

Sails were generally balanced between the masts. It would be unusual (for example) to set royals or t’gallant stays’ls on one mast only. (There are exceptions to this, especially when running dead before the wind, when some sails – particularly courses – might be set on the foremast only).

Besides setting different sails to account for different strengths of wind, the other main reason for setting fewer sails would be to simplify the rig to make maneuvering easier, when speed was not important or desirable. Coming off an anchor, coming up to anchor, maneuvering in a harbor, heaving to to launch or pick up boats, surveying an unfamiliar shore, or going into battle, were all circumstances where this might apply.

Typical sails set for maneuvering in these circumstances would be the top’sls, the spanker, and one or two jibs. In battle, in particular, the courses were rarely used. They were more difficult to trim for rapid maneuvers, they obscured the vision of the officers and helmsmen, and they were considered a fire risk, as they were too low and too near the matches and muzzle blasts of the cannon. So, in battle, if more speed was desired, the t’gallants might be set instead of the courses.

That should give a good general overview of how the array of sails was adjusted to allow for different conditions. More detail would go into the uses of reefing and some slight variations for different points of sailing in different wind conditions.

There are a serier of historical fiction novels by Patrick O'brien that explain tall ship sailing quite well. They are also fantastic reads. Most of what I am about to explain came from those novels and suppliments to those novels.

Assuming your talking about a 3 master frigate or ship of the line. There are 2 different types of sails, square sails and jibs, wich are triangular. In very simple terms a captain would set mostly jibs or triangular sails if the wind was hitting the ship in the sides. Otherwise square sails would be set if the wind was coming from behind the ship. As you go higher up the mast sails usually get smaller and used for lighter winds. Mainsails at the bottom of the mast were the largest but not always the most useful as winds weren't always strongest that low. The main top sails above them were generally more useful. They were low enough to be anchoured to the ship more securely and high enough to catch a higher stronger breeze. It gets really complicated but generally one would pick a combination of sails that allow for an even thrust across the ship without having any mast carry more thrust than the other 2. Though the foremast and the mainmast usually did most of the heavy lifting. Setting too much sail is also a major concern too as a sudden gust may carry away a sail or mast if too much sail is set. Its important to remember that the large sailing vessels built durring the era of sail were probably the most complicated machines that existed at the time and there are a million variables to consider. If you have any more specific questions I would love to try my best to help you out.