I have a feeling that most of the time people just went over the side. But, I have no idea if that is correct or what the exceptions are. Did size matter, the country the ship came from, what kind of cargo it was hauling, time period, etc.
It very much depends on the time period, but during the time I study, most European sailing ships would have a few "bathroom" options:
Go over the side, through a gunport or other port or just over the rail;
use a chamber pot and dump it later;
use the "heads" at the beakhead of a ship, which could be fitted with seats of ease (the square boxes on the Vasa here would have had seats at some point)
use a private head, often fitted into the quarter-gallery of a ship for officers to use.
