The literature of the mid-19th century Royal Navy is fairly silent on masturbatory practices, but we can assume that masturbation took place at sea just as it did anywhere else. As to the how, warships of the 1850s, like in earlier decades, had men sleep in hammocks, which would provide some privacy -- single-stall toilet or shower arrangements were not common until much later, around the turn of the 20th century, so your best bet would likely be to take care of business during your sleeping shift and attempt to be discreet.
We do know that mass copulation in ports (with women) was fairly unremarkable in ships of the Napoleonic era; there are amused descriptions of midshipmen, often 12 or 13, encountering 500 or 600 couples swiving away merrily at once on the lower decks of a warship, usually told with a knowing leer.