The most recent study about De Historia Piscium seems to be the one written by historian of science Sachiko Kusukawa (2000). While she presents in great detail how the book came about and how it was edited and printed, she does not provide explanations for this "flop of sales". So we will have to engage in some (informed) speculation.
One important thing to note is that the book was really expensive to make. Kusukawa calculates that the full cost came about to £360 for 500 copies. Calculated on 480 non-defective books, the cost of a single copy was 15 shillings (360*20/480). The currency converter at the British National Archives tells us that the total cost for publishing the book was roughly equivalent (in 1690) to 67 horses, or 86 cows, or 4011 days of wages (11 years) for a skilled tradesman, or £43,000 in 2017.
Almost two-thirds (£239, 66%) of the cost came from engraving the 187 copper plates. The relative costs of paper (£45, 12%) and printing (£57, 16%) were much lower compared to that. Less than half (£163) of the total cost of the Piscium book came from 70 subscribers. It included a donation of £63 by Samuel Pepys, then president of the Royal Society. While this money was meant to pay for the engraved plates - the subscribers got their names on the plates -, it did not even cover this specific cost: we can see here that the final cost was mainly the result of the decision by John Ray (the main author, who had continued the project initiated by the late Francis Willughby) to publish a book so lavishly illustrated. Willughby's previous posthumous work, Ornithologiae libri tres (1676, also completed by Ray), contained only 77 plates, paid for by his widow Emma. (Anachronistic note: Ray did not have to pay for image rights: he picked up images he found in previous works and asked artists to copy them).
The lowest price for the Pisicium book (subscriber price, worst paper) was £1 0s 8d (approximately 11 days of skilled tradesman; £123 in 2017). The highest was £1 8s (non-subscriber, best paper; 15 days of skilled tradesman; £168 in 2017). For comparison, Robert Plot's The natural history of Oxford-shire (1677), which contained 16 plates, was sold for 8 and or 9 shillings for subscribers and non-subscribers respectively. The price of Ray's fish book was twice to three times this.
And this is where the Royal Society's problems began. Its beautiful Piscium book did not sell at all. If we assume that 460 copies were put in the market (20 were given to Pepys) in April 1686, the Royal Society was still trying to negotiate a deal with an Amsterdam bookseller to take on 400 unsold copies in June 1687 (and this did not work: only 2 copies were sent to Holland). In the following years, the Royal Society used copies in lieu of cash (as salaries for some employees notably), or gave them away. Between 1688 and 1772, the RS received £111 for the book.
To put it simply, Ray and the Royal Society overestimated the appeal of a very expensive book about fishes. Ray was very pleased with the pictures and he believed that the "beauty and elegancy" of the engravings would attract buyers. This was a labour of love for him, resulting from the Grand Tour of Europe he had embarked on with Willughby between 1662 and 1666, collecting specimens, attending dissections, and buying books and images. His project had also religious underpinnings: Ray wanted to demonstrate, through the methodical classification used in the book, the existence of an uncorrupted universal language that allowed people to describe natural objects in an unambiguous way.
The abysmal market failure of De Historiae Piscium had actually a recent precedent: that of Robert Morison’s Plantarum historiae universalis Oxoniensis (Mandelbrote, 2015). Like Ray's Piscium, Morison's Plantarum had been a magnum opus for its author, meant to demonstrate a new way of classifying natural objects, in this case plants. Morison's herbal was rich of 126 plates. Meant to be published by the Oxford University Press (OUP) and paid by subscribers (whose coats of arms were printed on the plates), the book was started in the early 1670s and was plagued by delays and financial problems. One "Second part" was published in 1681 (the first volume was never written). It may have been published in 750 copies: 470 were never sold. Morison's accidental death in 1683 put the already ailing project in jeopardy. Money issues, copyright troubles with Morison's estate, and criticism of his classification method (by Ray among others) resulted in further delays and the publication of the last volume only happened in 1699. It was a large failure and "a contender for being the most disastrous" book ever undertook by the OUP: this ambitious project had consumed material and human resources for two decades, with little to show for it. John Ray, who was writing his own Historiae Plantarum in the mid-1680s, wavered over to include figures or not in this book but finally chose to print it (under the auspices of the Royal Society) without illustrations due to the cost.
These projects were ambitious and highly personal endeavours that were not well-thought-out in terms of development, financing, and marketing. We do not know why the wealthy public targeted by the OUP and the Royal Society did not buy those books: the hefty price was certainly a major point, but these people may have not been interested in fish or in a novel plant classification in the first place, even with nice pictures, and the marketing strategies of the publishers were not up to the task.
Sources
Kusukawa, S. ‘The Historia Piscium (1686)’. Notes and Records of the Royal Society of London 54, no. 2 (22 May 2000): 179–97. https://doi.org/10.1098/rsnr.2000.0106.
Mandelbrote, Scott. ‘The Publication and Illustration of Robert Morison’s Plantarum Historiae Universalis Oxoniensis’. Huntington Library Quarterly 78, no. 2 (2015): 349–79. https://doi.org/10.1525/hlq.2015.78.2.349.