It's from the 19th century and not the 17th, but Herman Melville talked about the subject a little in his novel White Jacket (in which he describes life aboard a Man-o-War).

Here one passage I found:

In men-of-war, night and day, week in and week out, two life-buoys are kept depending from the stern; and two men, with hatchets in their hands, pace up and down, ready at the first cry to cut the cord and drop the buoys overboard. Every two hours they are regularly relieved, like sentinels on guard. No similar precautions are adopted in the merchant or whaling service.

...

Next day, just at dawn, I was startled from my hammock by the cry of "All hands about ship and shorten sail!" Springing up the ladders, I found that an unknown man had fallen overboard from the chains; and darting a glance toward the poop, perceived, from their gestures, that the life-sentries there had cut away the buoys.

It was blowing a fresh breeze; the frigate was going fast through the water. But the one thousand arms of five hundred men soon tossed her about on the other tack, and checked her further headway.

"Do you see him?" shouted the officer of the watch through his trumpet, hailing the main-mast-head. "Man or buoy, do you see either?"

"See nothing, sir," was the reply.

"Clear away the cutters!" was the next order. "Bugler! call away the second, third, and fourth cutters' crews. Hands by the tackles!"

In less than three minutes the three boats were down; More hands were wanted in one of them, and, among others, I jumped in to make up the deficiency.

"Now, men, give way! and each man look out along his oar, and look sharp!" cried the officer of our boat. For a time, in perfect silence, we slid up and down the great seething swells of the sea, but saw nothing.

"There, it's no use," cried the officer; "he's gone, whoever he is. Pull away, men—pull away! they'll be recalling us soon."

"Let him drown!" cried the strokesman; "he's spoiled my watch below for me."

"Who the devil is he?" cried another.

"He's one who'll never have a coffin!" replied a third.

"No, no! they'll never sing out, 'All hands bury the dead!' for him, my hearties!" cried a fourth.

"Silence," said the officer, "and look along your oars." But the sixteen oarsmen still continued their talk; and, after pulling about for two or three hours, we spied the recall-signal at the frigate's fore-t'-gallant-mast-head, and returned on board, having seen no sign even of the life-buoys.

The boats were hoisted up, the yards braced forward, and away we bowled—one man less.

My friend /u/RMission wrote an article on this subject, in particular how people back then would be revived from drowning:

Resuscutation of Drowning Victims at Sea.

It will inform you of the subject more than you could imagine.

There were no formal rules in place and so the usage varied, as others have suggested, depending on service and the decisions of the captain. Any 'rule' you create to describe what occurred would probably be riddled with exceptions as well. Although there was no formal ruling on rank I'm fairly certain that a captain falling overboard would produce a much larger response than just a sailor.

Another complication would be weather and wind. Sailing boats, especially large men-o-war or fat heavy merchant ships, can be difficult to turn around or even just heave to (stop). Which is why the smaller boats would be sent back. But even stopping has considerable risks in certain seas, weather and/or locations.

So, a captain falling overboard from a warship on a calm day in a warmish sea with no war going would be far more likely to be saved than a junior sailor on a dodgy merchantship in bad weather off a lee shore on a hostile coastline.