I've seen differing answers, but how seriously did the average mariner take tales from writers like Ptolemy and Pliny of mythical creatures like the Whirlpool, griffins or stories of impenetrable fog, oceans boiling away, etc? Especially after Columbus landed in the Americas and the Portuguese rounded the Cape of Good Hope - it seems like having explored so much they would realize the improbability of them.
I’m not sure it is possible to answer your question completely because beliefs about sea monsters, fantastic land and sea creatures, and mythical races of people were so widespread during the period. Virtually every culture it seems had myths about sea monsters. Local beliefs were informed by local cultures, local practices, and local histories, which means that what I generalize about one place might be totally wrong about another. To make matters more complicated, most sailors left no writing, so we have to rely on captain’s sources, travel sources, natural history sources, and art, which doesn’t necessarily talk about individual sailors. But in general, I would answer your question with three points: First, there was certainly widespread beliefs in sea monsters, fantastic creatures, mythical beings, and other weather phenomena throughout the early modern period on both sides of the Atlantic. Second, many European sailors believed that these monsters existed (though not all), but accessing their voices in existing sources is difficult. Third, the understandings and conceptions of monsters changed over time.
For historians today, when it comes to sea monsters, it is impossible to know what exactly people saw, what events they exaggerated, what rumors they conflated as true, and what stories they simply made up. Knowing exactly what happened isn't actually that important at all. What matters is that people BELIEVED these things. For people in the early modern world, they lived on Earth alongside the divine. Theirs was a world in which the Devil and demons appeared before many individuals and actively participated in day-to-day life, as demonstrated by the many Inquisition cases on both sides of the Atlantic where people legitimately believed they had encountered, interacted with, and sometimes carried out the Devil’s bidding. Theirs was a world where God’s signs were written in the Heavens, where earthquakes demonstrated God’s disfavor, where the Ancients knew many mysteries of the world that had since been lost, where indigenous people in the New World could shapeshift into animals, where lead could be turned to gold, where swarms of locusts could be exorcised, and where medicines called for dragon’s blood and unicorn horn. Magic, angels, spirits, and occult forces were as much a part of daily life as the weather.
According to Anthony Grafton in his book New World, Ancient Texts, Europeans reconciled the shock of discovery to ancient texts with relative ease. They mixed new discoveries into ancient sources without completely disregarding the ancient authorities. Therefore, descriptions of monsters remained common in intellectual sources and folktales, even as travelers, sailors, and traders explored the globe. Early modern natural histories are filled with descriptions of monsters and dangerous beasts alongside real animals from the Old and New Worlds. Take for example one of the most famous early printed books, the Nuremberg Chronicle, which includes a famous smattering of the strange races of the world (the images start on page 69). Botanical and natural scientific texts contained mixtures of real and imagined creatures. Conrad Gesner’s work, which pulled from earlier sources, contains descriptions of many real animals alongside various fictional ones, including sea monsters. Aside from natural history texts, maps were also filled with images and descriptions sea monsters. These creatures were often depicted as mashups of land animals because, according to Chet Van Duzer in his book Sea Monsters on Medieval and Renaissance Maps, since the time of Pliny, there was the belief that every land animal had an equivalent sea creature. Van Duzer argues that sea monsters on maps played two roles: “They may serve as graphic records of literature about sea monsters, indications of possible dangers to sailors...Second, they may function as decorative elements which enliven the image of the world, suggesting in a general way that the sea can be dangerous, but more emphatically indicating and drawing attention to the vitality of the oceans and the variety of creatures in the world, and to the cartographer’s artistic talents” (11). Captains and sailors were often familiar with these texts, maps, and descriptions of sea creatures, even as knowledge of the natural world expanded.
Many scholars (like Gesner) were at least a little skeptical of the existence of such beasts, especially as empirical practices came to be regarded as the most important discerner of truth, but sailors continued to believe that the world was filled with many unknown, dangerous, and benign sea creatures. For example, Columbus, along with many other sailors throughout the fifteenth and sixteenth centuries, wrote about seeing mermaids. In Columbus’s case, historians today believe his mermaids were actually manatees that he encountered near Hispaniola. Henry Hudson described his crew’s encounter with a mermaid in 1608: “This morning, one of our companie looking over boord saw a Mermaid, and calling up some of the companie to see her, one more came up, and by the time shee was come close to the ship’s side, looking earnestly on the men: a little after, a Sea came and overturned her: From the Navill upward, her backe and breasts were like a womans (as they say that saw her) her body as big as one of us; her skin very white; and long haire hanging down of colour blacke; in her going down they saw her tayle, which was like the tayle of a Porpesse, and speckled like a Macrell” (quoted from Ellis, Monsters of the Sea, 79).
Both Vespucci and Magellan wrote about encountering giants on their travels. Vespucci described meeting with giants in Brazil: “We found a village about twelve houses where we met only seven tall women. They were all one-and-a-half palmos bigger than me...Then thirty six men arrived and entered the house where they were drinking and they were so tall that even kneeling each one of them was still bigger than me upright. In short, if we refer to their stature and to the proportions of their bodies, they were giants” (quoted from McEwan, Borrero, and Prieto, eds., Patagonia: Natural History, Prehistory, and Ethnography at the Uttermost End of the Earth, 128). Magellan’s crew encountered indigenous peoples near the tip of South America, probably the Tehuelche, and believed them to be giants. His sailors described them as twice as tall as a man. They supposedly captured two of them and attempted to bring them back to Europe; unfortunately, the giants did not survive the journey.
Stuart B. Schwartz writes in his book Sea of Storms that over the course of his time in the Caribbean, Columbus started to have an understanding of hurricanes. Late in his career, his ships were supposed to sail back to Spain along with vessels commanded by one of his many rivals. Based on the clouds and temperature, Columbus predicted that there would be a great storm, but others in the convoy did not believe him. Columbus stayed in port to wait out the storm, while the rest of the fleet set sail back to Spain. The entire convoy was destroyed with hundreds of lives lost, while Columbus easily rode out the storm from the safety of the harbor. Almost immediately, rumors swirled that Columbus himself had called down the storm onto the fleet. In another case, James Sweet in his book Domingos Álvarez, African Healing, and the Intellectual History of the Atlantic World uses the African healer Domingos as a case study. During one episode, Domingos was taken aboard a slave ship sailing from northeast Brazil to Rio de Janeiro. During the voyage, the wind completely died, and the ship sat adrift for days. Domingos, rumored to be a powerful healer, somehow caught the attention of the crew, who discovered charms and amulets in his clothes. They tied him to the mast and whipped him mercilessly, suspecting that he had cast a spell that stopped the winds. After Domingos's charms had been burned, the wind soon picked up again. Upon reaching Rio, the captain of the ship denounced Domingos to the Portuguese Inquisition. Both of these examples show that in addition to mythical creatures, sailors also believed that people could summon storms through sin, by violating superstitious practices, or through sorcery.
Sailors wore magical amulets, said prayers, and carried out masses on board ships to protect themselves from monsters, storms, and drownings. Death seemed to lurk in every swell and storm. The unknown depths below them were filled with strange and unknown perils. A whale or something large floating in the water when seen at a distance might have resembled a sea monster. An oarfish or a giant squid may legitimately have seemed like sea serpents. Sailors’ fears mixed with active imaginations of bored, tired, and uneducated sailors to produce all kinds of rumors and stories. Travelers often relied on these stories and rumors to describe the New World and its creatures. Descriptions of monsters ended up in many of the travel sources and natural history texts based on no more than hearsay. For example, Columbus wrote that he never saw any human monsters like those shown in Gesner’s text, but he heard from indigenous people that they existed elsewhere. Columbus couldn't prove the negative, so he mentioned that the rumors appeared true, though he did not see them personally.
One of my favourite Askhistorians answers ever addresses at least part of your question: Would a crew of a ship departing from Venice in the 14th century towards Middle East have any plans ready in case they encounter a leviathan on their route?
Original answer by /u/sunagainstgold.