There have been periodic accusations by the press of mental instability of candidates from nearly the beginning of the Republic, which would start as campaign talking points but in the opposition press would continue throughout someone's administration. (It's also worth noting that historic definitions of mental illness are somewhat different than our modern version, but that's an entirely different topic that I won't address in this answer.)

One prime example is Harry Truman going off on music critic Paul Hume of the Washington Post. As I explain, though, for that particular incident there were multiple mitigating circumstances: the sudden death of Charlie Ross, which meant the long time system Truman had used for blowing off steam - writing angry letters he never sent - went haywire that particular day more than anything else.

But the two that really jump out are Franklin Pierce and Herbert Hoover.

Pierce had been out of politics for a number of years and had assured his wife he was done, but got nominated on a very late ballot as a dark horse candidate in 1852 and won fairly handily. As President-elect in January 1853, Pierce, his, wife, and their son were still in New Hampshire. After spending Christmas and New Years in Andover, they took a train back to the family home up in Concord to pack for the White House.

Unfortunately, it derailed and went straight over a 15 foot cliff. Miraculously, there was only one fatality among 60 passengers - but that happened to be his son Benny who had stood up to look out the window in the seat in front of his father (who successfully held on to his wife next to him, but tried and failed to grab his son as the train went over the cliff), and when their car flipped over, the window hit a rock and partially decapitated him. The tragedy was compounded by the earlier, if less horrific, deaths of their other sons.

His wife was genuinely devastated - Jane Pierce didn't hold social events until a year later, chose black for her inaugural dress along with draping the White House in it, continued talking with her dead son in both seances and in every day life, and was referred to as the "Shadow of the White House" by opposition press - but Pierce was a mess too. He chose to affirm his oath of office on a law book rather than swear it on the Bible since he reputedly felt God was punishing him for getting back into politics and breaking his word to his wife, and while he always had a reputation as a drinker (he eventually died of cirrhosis), if he wasn't an alcoholic before, he almost certainly became one during his term. Afterwards, there is a possibly apocryphal story about an answer he gave about what he planned to do after his term: "After the White House, what is there to do but drink?"

The other that comes to mind is Hoover, who as Eric Rauchway and I briefly discuss starts seeing mortal enemies everywhere, even among his erstwhile allies; one of the break-ins he orders is of fellow Republican Hiram Johnson. As Rauchway mentions, this is less clear cut as it's genuinely hard to separate out what was political versus paranoia as Hoover took things incredibly personally, but this is a man who at one point during the Bonus Army struggle literally has padlocks placed on the White House gates. His behavior and apparent comments during this makes it pretty clear that he is not just locking the protestors out but locking himself in.

There's also late term Nixon, but that's much more complicated. /u/restricteddata has discussed some of the implications around his behavior here and here among other places.