The answer is a mix of both. I'll only really speak about the military here because its the area I know more about. But its probably fair to assume that much of this could be said for merchants as well.

Formal officer training existed as early as the late 1600s in England, those aiming for a career in the navy would enter the service as a midshipman, either through recommendations, purchasing commissions or occasionally talent.

From there they would learn the basics necessary to be an officer (map reading, compasses, basic navigation and maintenance) while aboard ship, often learning from more senior officers. Those who wished to claim higher in the service beyond the rank of captain would then sometimes attend more formal training colleges (in the UK Greenwich is probably the best example of this, although it didn't open until 1872)