I've written about this kind of thing before! You can find some older answers here and here.

Briefly put, the position of master grew out of the structure of the Royal Navy around the time of the Spanish Armada and leading into the Anglo-Dutch wars, in which case a military person of some sort, often a noble, would be in charge of a ship, but that person may or may not know how to sail or navigate a ship. The "gentlemen captains" were often seen in contrast with the "tarpaulin" sailors who sailed the ship, and there was a general divide between soldiers on the ship and the sailors who sailed it. People who were appointed captains of ships were often appointed on the strength of their background or family. In 1676 Charles II took the decision that ships could take on boys (not older than 16, and presumed to be gentlemen) with the goal of having

'the families of better sort among our subjects to breed up their younger sons to the art and practice of navigation in order to the fitting them for further employment in our service at sea ... ' (via Rodger, Command of the Ocean).

This sounds reasonable, but the language is a major break with tradition -- in the language of the day, "arts and practices" are skills, which are learned by craftsmen or artisans; gentlemen are gentlemen precisely because they are not required to have skills. The idea of a qualifying examination for a commission, which was codified in 1677, was socially revolutionary; although Pepys attempts to take credit for it, the admiralty in general had to convince Charles II of its necessity.

In any case: These "kings' letter boys" became the midshipmen of later days, and skill in navigation was required to be an officer.

That said, the sailing master was the warrant officer who was ultimately responsible for the ship's navigation; the position was crucial both for training future officers (midshipmen were master's mates, that is, mates to the sailing master) and because even with a qualifying examination, some midshipmen wanting to be lieutenants skated by with a friendly board of examiners. As a warrant officer, the sailing master would be socially placed with his messmates, the other warrant officers, such as the boatswain, gunner, carpenter, surgeon, chaplain, purser and the like.

In the context of the movie, which is based largely on the book The Far Side of the World, Allen is chosen as master because of his familiarity with whaling and the South Seas fishery in general.