Which O'Brian is surely the great writer of the world, and he's correct that captains had fairly wide latitude to change the appearance (both external and internal) of their ships. We tend to think of the "standard" Royal Navy ship to be painted to resemble the current HMS Victory, itself the great ark of the world, in what's been termed the "Nelson chequer." The paint scheme there is of buff yellow (ochre) and black stripes, with the gunport lids painted black to form the "checkered" effect. But that's a scheme that was not necessarily widely adopted in the Navy, and in fact when launched Victory was simply varnished with a bright red and blue frieze around the top of the hull. Later it was pained buff overall, and Nelson altered the ship's color scheme when he was appointed admiral into the ship in 1803.

Interestingly enough the ship was repainted in black and white in 1816, and when efforts were made to "restore" it in the early 20th century, the paint applied was a very bright yellow and black -- research undertaken later showed that the original yellow was probably simply an ochre color, closer to what it is today. The ship is still probably too orange -- research into Nelson's orders regarding the paint before Trafalgar says that it was white mixed with yellow at a ratio of 6:1 -- and it may be closer to what a French warship of the time period would have looked like, based on the records we have from e.g. Walter Fawkes after the battle of the Nile in 1798.

Edited to add: I went down a bit of a rabbit hole on this, and apparently part of the reason for the black-and-white color scheme is that when ships were laid up "in ordinary," that is, stored in harbor and not actively in use, they were painted white overall to reduce wear on the ship's timbers, white paint being reflective. Then if they had to be taken out of ordinary, the dockyard would paint the gunport lids black and use black paint to divide up the broadsides. End edit

Victory's current interior bulkheads are varnished and stained, but we know that sometime between 1803 and 1805 they were whitewashed, replacing the previous red paint that they had had since at least the Battle of St. Vincent in 1797. Victory's carpenter in August of 1805 recorded "To whitewashing the Tiers, Wings Cock-pits Store-roomes (sic) Lower & Middle Decks – Lime 12 Bushels Glue 12 Pounds." The whitewash was meant to improve lighting in the lower decks, as well as to kill bacteria, as was common in outbuildings and outhouses.

We do know that Nelson asked his captains assembling under him to paint their ships in a similar color scheme, as well as to paint their mast-hoops, to aid in recognition in battle when smoke would make identifying ships difficult. So your hypothetical frigate-captain serving under Nelson would have not had much latitude there!

But let's consider captains on other stations, or independent commands -- here we have much more latitude to undertake their own paint schemes, or to decorate the ship with gilding or other types of accent work. If the captain's admiral had not given orders to the contrary he could paint the ship in pretty much any manner he pleased -- the Admiralty from time to time gave orders for how ships should look, but they were widely ignored. As far as other alterations, the West Indies station was notorious for having captains who would rake their masts to stand straight up; this gave the ships a fine appearance, at the loss of speed and performance. A captain wouldn't go so far as to add an extra mast or anything odd like that, but he would have wide latitude to experiment with spars and yardarms, which gave us the Bentinck shroud and Bentinck booms, for example.

Moving on to your questions about the ship's complement -- each ship had an established complement of men, and many carried a few extra men, boys, or even women, on their books, but not in large number -- the ship's purser and captain were responsible for victualling the ship's complement, and would have to have authorization for supernumaries. (The opposite practice, of carrying fictitious names on the ship's books for their victuals and money, was illegal but was sometimes used to build "sea time" for young officers.) The list in Lavery's book is certainly prescriptive -- I wrote about this more here,:

Pepys' goal in the Establishment of 1677 to draw up a "solemn, universal and unalterable" classification was mostly about budget -- if he knew how much manpower was needed for each ship, and how many ships were in each rate, he could much more precisely budget for victualing, and more crucially plan victualing ahead of time so that meat and beer wouldn't spoil before being put up (and ensure that sufficient quantities of provisions could be delivered to the victualers without perverting the market). Pepys' classifications were minute in detail: seven men were allowed to a 42-lb gun, five to a 32-lber, four for an 18-lber, etc; or, put another way, half a gun crew, or, put a third way, the number that would allow the ship to fight one broadside at a time (a sail-trimmer was part of the gun crew).

Now, Pepys' classification scheme, although famous for providing us with the First through Sixth rates and unrated ships, wasn't an attempt to standardize classifications but merely manning requirements. What was happening instead is that if the Admiralty heard from ship captains that "hey, this ship is hale and weatherly, let's build more like it," and ordering the same type of ship, broadly speaking, from the yards. Much of the idea of "classes" that we get later is somewhat retrofitted; we can speak of a "class" of single-decker frigates mounting 9 lb guns and some amount of length at the waterline, but they were never intended to be exact copies of one another.

Regarding this:

To what extent were shipboard positions hired by the captain themselves or assigned to the ship by the navy?

A captain had a fairly wide latitude to bring "followers" with him from ship to ship, usually his coxswain, servant, midshipmen, and sometimes other warrant officers, but the port-admiral or other person manning the ship would generally assign lieutenants to the ship, though a well-thought-of captain could request specific men. The permanent officers assigned to a ship were the boatswain, gunner, and carpenter.

(References to the ratio of paint and the carpenter's records are from "The Application and Scheme of Paintworks in British Men-of-War in the Late Eighteenth and Early Nineteenth Centuries", published in The Mariner's Mirror, July 2013.)