Most military vessels of the late age of sail had a number of marines on board. The marines were in place to act like a police force, they guarded many areas of the ship where sailors could cause trouble(powder room, food stores, grog, etc.).

Next, the punishment for mutiny was death, you'd being hanged from a yard arm. Discipline was strictly enforced, major infractions would result in corporal punishment, like being lashed up to a grating and whipped with a cat-o-nine tails. Lazy or slower sailors could also be "started" which entailed a wack on the back with the bosun's cane.

The most well known ship mutiny occurred 1789 aboard the HMAV Bounty. The Bounty did not have any marines and only one commissioned sea officer. Accounts of the crew say that prior to the mutiny Bligh made very strange decisions and had an unpredictable explosive temper.

Mutiny on the HMS Hermione resulted in the the killing of it's captain and all commissioned officers. The Captain was incredibly brutal often ordering that the last man down from the tops be flogged. Three topmen felt to their deaths after racing down in order to avoid punishment. The captain threw their bodies overboard without ceremony.

Large mutinies did also occur for the Royal Navy during the late 18th century. The Spithead and Nore mutinies were some of the biggest mass mutinies involving numerous RN ships. The overall cause of the mutinies were the poor pay sailors received.

To answer your question how a captain would prevent mutiny is pretty simple. Keep discipline with the thread of punishment, have marines, pay your sailors, and don't be brutal/insane. Interestingly, demands from the Spithead mutineers did not including cessation of flogging. They understood the need from punishment to keep order, sailors just didn't want be commanded by the kind of tyrant that Hermione had.