Try as I might, I cannot find an actual source for this. Various books cite each other, but I can't find anything primary. For instance this cites The Heretic's Feast, and it's true that that book mentions this anecdote, but the book itself does not give a source, so I'm not inclined to buy it.
There is evidence to suggest this did not happen since Rene did not have a wife - He never married. However He did have a child with Helena Jans van der Strom, so maybe it was Her dog that He nailed?
Did Rene Descartes nail his wife's dog to a board?
The short answer is no he did not.
Did someone else nail his wife’s dog to a board?
The short answer is no.
So what is the basis of this story?
Peter Harrison suggests that Richard Ryder in his Animal Revolution (1989) mistakenly attributed the story to Descartes. Instead, it was Claude Bernard, a nineteenth-century French physiologist, who became associated with vivisecting dogs. Bernard’s attitude towards experimenting on animals was to disregard the pain and suffering they showed. He argued that this disregard was an essential feature of being a ‘proper scientist’. One version of the story associated with him recounts how upon his wife (Marie Francoise “Fanny” Bernard) and daughters returning home they found the family dog being vivisected by Bernard. [This story is mentioned in Mary Midgley’s Animals and Why They Matter, but I cannot find the piece she references, so any help on checking it would be helpful for the sake of completeness]. However, a contemporary journal (Zoophilist, which supported anti-vivisectionism) was unable to verify this story. Their version of the unverified story was also somewhat different. It was Bernard’s daughter who was looking for the family dog, and while doing so she found her father carrying out a live vivisection on her pet. Once again, this story was never verified even by an anti-vivisections journal. What remains true though, is the fact that Claude and Fanny separated around 1870, and Fanny became an avid anti-vivisectionist raising their two daughters with the same sentiment in mind.
So why did Descartes get a bad reputation?
To put it in a concise but incomplete manner, he argued that animals were machines/automata and that they did not have self-consciousness. The combination of these two statements are often interpreted as a denial of animals having feelings. However, Harrison pointed out that Descartes distinguished between sensations of bodily organs and conscious sensations. For instance, when you are sleepwalking you are not technically conscious of your sensations, but you are still navigating by relying on some of your senses. So within the system of argumentation set up by Descartes animals had feelings but they were not conscious of it. Or their feelings did not arise from self-consciousness as in the case of humans. Ultimately, Harrison’s main statement in his article is that Descartes merely pointed out that within the established system of reasoning about animals, there were ‘no irresistible reasons for asserting’ that animals had feelings.
To end on a more positive note: Descartes actually had a dog called ‘Monsieur Grat’, aka Mister Scratch, and they used to go on walks together.
Sources:
Mention of Monsieur Grat - Jack Vrooman, Rene Descartes: a Biography (New York: G.P. Putnam's Sons, 1970), p. 19. - https://archive.org/details/renedescartesbio00vroo
Sinding, C. (1999). Claude Bernard and Louis Pasteur: Contrasting images through public commemorations. Osiris, 14, 61-85. - https://www.jstor.org/stable/301961
Harrison, P. (1992). Descartes on animals. The Philosophical Quarterly (1950-), 42(167), 219-227. - https://www.jstor.org/stable/pdf/2220217.pdf
Richard Ryder (1989). Animal Revolution: Changing Attitudes Towards Specieism - for the exact quote on Descartes, see page 57.
Mary Midgley (1998). Animals and Why They Matter.
Article about the story not being verified in the Zoophilist: https://books.google.co.uk/books?id=PLR3jSu0bu0C&newbks=1&newbks_redir=0&pg=PA85#v=onepage&q&f=false