A couple of points:

The 'Mammal vs Fish' debate had been a fairly passionate debate among the circles of whalers, scientists, and other related occupations in and before Melville's time. An excellent book, Trying Leviathan, examines the case of Maurice v. Judd, a New York court case regarding the status of a whale as a fish (stemming from laws pertaining to fines and fees on fish oils). This occurred around 1818/19, only a few decades prior to Moby Dick being published.

Trying Leviathan examines the very different and unique perspectives that those in occupations relating to whales had on their classification status. It should be noted that what Melville writes isn't necessarily a reflection of his own beliefs - rather his character Ishmael, a whaler. His brash rejection of scientific authority was pretty common among the deckhands and skippers of whale watching vessels, according to Trying Leviathan . He throws out the rigid science out of spite to their authors' lack of 'real experience' dealing with whales. He opts instead for the writings of an actual whaling captain (whose name escapes me), comforted by his positional authority in the seas rather than learned and abstract logic in academia. His dismissal of Linnaeus isn't necessarily because of the Swedes' conclusion, but more so the fact that the leading taxonomist had likely never caught and harvested a whale in his life, certainly not as Ishmael had done.

I have seen it referred to as Folk-biology, or in this specific case, 'Biology from the Mast', the conflict of a perceived 'absent judgement' on the status of a field that has been in practice for centuries. In this case, whalers had various reasons to contest Linnaen and related taxologist assertions that fundamentally changed aspects of their livelihood. Some might be financial, as I mentioned before regarding Maurice v. Judd having its roots in oil classification and fees that came with importing and stowing fish oils. Some might be cases of pride, why would a captain who has spent years of his life under the natural assumption that a whale was a fish change his mind so suddenly because of some Swedish scientist? These are general examples, but reflect the fundamental conflict over what might seem to be a silly argument.

We also need to note that when we ask if a whale is a fish, we're ultimately not even talking science. If you wish to dive into specifics, whale and fish are barely scientific terms. They're large and hopelessly nonspecific terms we use to group certain animals together by how our culture determines. If a fish is simply a creature that lives in the oceans, and whales do not live on land, then why can't a whale be a fish? Yes, we know that they are not biologically related, but in those terms we don't really discuss biology, because the words fish and whale are vernacular expressions that lump hugely unrelated groups of creature together for convenience. If we separate whales from fish due to their mammalian factors, do we further divide the category of fish? How do we explain 'non-fish' creatures so named, like the cuttlefish? It may seem daft that the idea of a whale as a fish was accepted for so long, and even after classifications that separated the two, but (relating back to Folk-biology) its also a bit incoherent trying to argue and draw scientific conclusions out of non-scientific words, if that makes any sense?

Finally, and I'll spend a brief amount of time on this, Melville's sections on cetology, specifically Ishmael's categorizing of whale species, is more often seen as a form of satire and literary meta reference (you'll notice he divides species into portfolios, for example). As well, a great interpretation that's become very common regarding the entirety of Moby Dick is that the book about the hunt for a white whale is in itself a White Whale. It is a long, perilous journey rife with moments of excitement (his vivid climax and poetic descriptions) and long periods of lost trudging through seemingly endless seas (his long, dry, chapters regarding whales and whaling, the very length of the book, etc.) that the reader goes on. Therefore, sections like those arent necessarily meant to be read in a literal sense. This is more in the field of literary criticism and less in history and culture, so I'll leave it there.

Hopefully I've helped out and made some sense of this, let me know if you have further questions?

Taxonomy is mostly about, well... lineage, I guess. Fish and whales evolved apart a very, very long time ago, while whales and, for example, primates, share a more recent common ancestor, so it's generally accepted that whales and primates are all mammals, while fish are distinct. (Gills, milk, and all that aside, the evolutionary origins are what determine what species belongs to what phylum/genus/etc.)

The importance of lineage in this case is only apparent once you know about evolution and about how living organisms' information is encoded in DNA. Either that, or you've heard it on TV. In any case, "whales live in the water and have fins, so they're fish." is the predominant, and pretty obvious, attitude of anyone who isn't specifically informed on this topic.

Whether Melville actually believed this or was just paying attention to detail in dreaming up his uneducated fisherman Ishmael can't be said for sure, but the idea that whales aren't fish is really not apparent whatsoever unless you read Discovery Magazine as a kid, watch lots of nature documentaries, or asked the right stupid questions in biology class. Moby Dick is set in a time where a half-decent education was as big a privilege as eight-digit incomes are today, and Ishmael is a fisher.

This is not intended to be a comprehensive answer, rather an addendum to /u/an_ironic_username's comment, solely to explain that it's a privilege for you and me to be fully aware of the distinction between marine mammals and fish. Please excuse any minor factual inaccuracies, I hope the point got across anyway.