Seamen are some of both the most important and most overlooked classes of laborers in American (and world) history; hugely important to the labor movement, the settlement of the Americas, the Columbian exchange, you name it - and thankfully, we have a huge haul (pun not intended) of direct testimonials, court records, diaries and oral histories to draw on when looking at the lives of seamen.

I'll be referring mostly back to Between the Devil and the Deep Blue Sea - Merchant Seamen, Pirates, and the Anglo-American Maritime World, 1700-1750 by Marcus Rediker, most specifically chapter 2, 'The Seaman as a Collective Worker.' Really great read.

Sailors spent most of their time working, to be frank. Work on large ships was very taxing and occurred essentially around the clock, with minimal downtime. Especially in the early period, basically everyone chipped in to make sure that the ship ran smoothly and got to port quickly, as any time wasted would result in lower revenues. 'The Watch' is a great example of how seamen worked:

Half of the crew was assigned to the starboard and half to the larboard watch. The captain supervised one, the mate the other. On the largest ships, the first and second mates took responsibility for a watch. Each watch served four hours on duty, then four hours off, alternating in work shifts (also called "watches") around the clock . The dog watch, between 4 and 8 P.M., was subdivided into two-hour shifts. This produced a total of seven shifts, ensuring that a watch would not work the same hours each day. Each sailor alternately worked a ten- and a fourteen-hour day. The starboard and larboard watches were the essential cycle groups on each ship. Their major responsibility was to guarantee the continuity of keeping the vessel running and true. Everyone made a roughly equivalent contribution by helping to keep the ship on course at the highest possible speed (pg 88).

While we may have the expectation of plenty of free time, work was central to the lives of all seamen, and was in reality a gigantic part of the 'social order' on ships -

One of the central features of seafaring work was its social visibility. Work was a public activity, so public in fact that any seaman, even when off duty, knew what work was being done, and by whom, by the distinctive yell each tar gave during his various exertions. Crews were extremely sophisticated in judging the quality of each man's contribution to the sailing of the ship [...] further, work was closely scrutinized since the collective well-being often depended on it. (pg 95).

Trading vessels and large sail ships exhibited something fairly unusual for the period - a significant division of labor. Specialization was extremely necessary for workers above the level of 'Jack Tar' (the common seaman) and included roles like the Carpenter, Surgeon, Cook, etc. While everyone was worked hard, Jack Tar tended to be worked the hardest -

Even when off duty, Jack Tar was never far from work. Anytime , anywhere, he might hear the mate's fearful cry: "up every soul nimbly, for God's sake, or we all perish" (pg 88).

As to your question about coming into port, Seamen during the period had a high degree of independence and negotiating power thanks to the dangerous and often deadly conditions on ships. When cargo ships came into port, and the cargo had been loaded wherever it needed to go, seamen often went into port to enjoy themselves -

Once free of command, seamen were footloose in the port towns, "rambling to and fro about the Country," as one disapproving captain put it" (pg 104).

Source: Between the Devil and the Deep Blue Sea - Merchant Seamen, Pirates, and the Anglo-American Maritime World, 1700-1750 by Marcus Rediker

Well, the ships didn't sail themselves -- the sailors on board had plenty of work to do and not much leisure time for the most part. I wrote about this here before, so that link should give you an idea of what people did at sea.