The twentieth century saw a variety of ways in which bananas were considered humorous, from the famous 'slipping on a banana skin' pratfall in physical comedy, to the humorous song 'Yes! We Have No Bananas' (which was likely an influence on the title of the Woody Allen film). Given this, it's not surprising that the craziness of all those crazy comedians with their bananas evolved to mean 'crazy' in a less humorous way. Looking on Google nGrams, the earliest example of the phrase 'going bananas' (which is a better search term than simply 'bananas', considering there's plenty of discussion of the fruit) I can find is in the Hawaiian Paradise Of The Pacific magazine, from 1964, which uses the phrase as a punny article title for an article about recipes involving bananas. The Oxford English Dictionary documents it as current slang at the University of South Dakota in 1968-1970, and Liza Minnelli says in an interview with The Times of London in 1970 of a regimented all-women hotel that 'I went bananas!' This suggests that it was already reasonably commonly being used in conversation at this point. I suspect that the popularity of the term also received a big boost from the 1971 Woody Allen film Bananas, a farcical comedy set in a fictional 'banana republic' (when asked why the film was called 'bananas', Allen famously said 'because there's no bananas in it!', referencing the 1922 song).

In terms of 'going nuts', Oxford English Dictionary documents an earlier meaning of the term 'nuts' meaning 'enthusiastic about', which goes back to 1785 (e.g., "I'm nuts about bananas!") The enthusiasm of this original meaning evolving to become insanity seems to be an early 20th century Americanism. The earliest use of the term 'nuts' to mean 'insane' in the OED is in a comic strip in the San Francisco Examiner in 1908. The earliest example of the term on Google nGrams is from 1917, in a story in the short story fiction magazine The Green Book Magazine. Judging by Google nGrams, the term seems very common indeed by the 1930s, with it being mentioned in Collier's, Time, Esquire and other magazines.