In the West, cranes called sheers were used. These tended to be light wooden structures, fixed at the base. They were very simple, consisting of a few long beams. They were mounted either on masonry structures, as can be seen in the Copenhagen Mastekranen, or on old ships. Ships that carried sheers were known as sheer hulks. The cranes on sheer hulks could be smaller than land based ones, as the effective height of the mast could be increased by tilting the ship. This was done either by shifting ballast, or by physically pulling the sheer hulk over using ropes and teams of men. The collections of the National Maritime Museum include a model of a sheer hulk masting a frigate, which can be seen online here

It isn't a tallship, but I actually helped raise the topmast on the Dove docked in Historic St. Mary's City, MD.

Ships were made at the time so that they could raise or lower their masts in case of inclement weather so that they could minimize the risk of capsizing.

On the deck there is a large windlass. Imagine a six-foot-wide, maybe two foot diameter barrel perpendicular to the bow. It has several slots in which to attach a lever, you have to remove and replace the lever as you move the windlass. The huge machine works as a gigantic winch. On the Dove, it was placed just aft of the foremast.

A line runs from the windlass, over the top of the mainmast's platform (you might think of it as a "crows' nest"), through the cap (on top of the mainmast, where the topmast will be attached), through an opening on the bottom of the platform. The line is then attached to the topmast, looped around the bottom then the top.

At this point you are ready to hoist the mast in place. Once the top reaches the platform above the mainmast, you undo the knots at the top and fix the line to the cap. You then attach the rigging to the top of the topmast (it's attached to a ring and attached to the platform ahead of time). After a bit more hoisting, you add the flagstaff and whatever else goes up there, hoist some more. The topmast is fixed in place with a wooden block that goes through a "fid" in the bottom of the topmast and holds it above the platform.

I made a pretty extensive PowerPoint about it full of my own pictures. If anyone knows how I can upload it publicly, I'm willing to share.

Edit: I should add that the Dove is a replica of a standard 17th-century vessel.

I should also maybe mention if it would be interesting to you that the tradition of converting older ships into crane ships did not stop after the age of sail. The USS Kearsarge famously was converted into a crane ship in the 1920s. Here is a before and an after photo of her.