How did whaling ships pull up whales next to them or onto the ship without capsizing?

The economic logic of whaling changed somewhat with the species of whale, but the longest-distance voyages and the most famous type of whaling ship sought sperm whales. Sperm whales have a large cavity in their heads filled with spermaceti, a clear-to-white, waxy material that had a variety of uses in candles, cosmetics, and oils. Sperm whales are also covered in a thick layer of blubber, which can be rendered into oil, and may contain ambergris, which can be used in perfumes. (Other types of whales were hunted for baleen or other products.)

Dead whales -- including those that have been killed with a lance, the normal method in 18th century whaling -- will float, so the general practice was to use whaleboats to tow the dead whale back to the whaling ship, where it would be fastened to the ship with a line (rope or chain) around its tail and another around its head or secured to its jaw. The head of the sperm whale would be broken open and the spermaceti removed with a ladle, after which whalers would use long knives to cut a length of blubber from the animal. That could be hooked onto one of the ship's yardarms, and as it was raised, continuing the cut would result in long peels of blubber being removed from the whale. When the whale was upside-down, its guts could be rummaged for ambergris, and when the blubber was eventually removed (it was boiled down into oil in large pots on the ship), the corpse of the whale would be cast off to sink.

If you don't mind the Wikipedia illustration, this image gives an idea of how blubber would be removed, and the other marine life that could be attracted to the process.