The 1935 definition in Albin Jay Pollock's The Underworld Speaks (apparently published by the FBI to help people spot gangsters by their speech) is:
He's bananas, he's sexually perverted; a degenerate.
This may be alluding to bent, the shape of a banana.
Bent is 1914 US criminal slang meaning dishonest or crooked, and 1930 US slang meaning illegal or stolen.
The eccentric, perverted or homosexual meaning of bent may be originally UK slang; it appears in 1930 in Brophy and Partridge's Songs and slang of the British soldier: 1914-1918 meaning spoiled or ruined. It soon after appears in 1942's The American thesaurus of slang: a complete reference book of colloquial speech in the definition for eccentric: "Balmy, bats, bent, [etc.]".
Another 1833 US slang meaning of bent is to intoxicated with alcohol or drugs.
The 1935 bananas is in brackets in the OED, so they're not convinced it is the same meaning.
Etymonline says the crazy meaning is much later: 1968. This year matches with the OED's third quotation from the University of South Dakota's Current Slang:
Bananas, adj., excited and upset; ‘wild’.—College students, both sexes, Kentucky.—I'd say it, but everyone would just go bananas.
The OED's second quotation is from a 1956 Ohio newspaper caption:
We heard the police broadcast!! They say you're bananas!!
But it's hard to gauge the exact meaning without seeing the picture.
Edit: I found a possible example of crazy bananas earlier than 1968 in The Spokesman-Review (Jun 22, 1962):
I refer to the taunt, suspenseful, real-life drama NBC brought us from Oakmount Country Club over the weekend - the National Open. Compared to it, Bonanza is bananas, and Dr Bon Casey is just another pill-roll.
This Ngram suggests this meaning really took off in the early seventies:
Turning to slang dictionaries, The American Heritage Dictionary of Idioms (1997) says of go bananas:
According to the lexicographer J. E. Lighter, this expression may allude to the similar go ape, in that apes and other primates are closely associated with eating bananas.
However, The Concise New Partridge Dictionary of Slang and Unconventional English (2007) says bananas (madly excited; mad; behaving oddly) is from 1957 and derives from banana oil (nonsense; persuasive talk) from 1924.
Contemporary synonyms are horsefeathers and appleseauce. The origins of banana oil are also unknown, but The American Heritage Dictionary of Idioms (1997) says it's possibly variation on snake oil (quack medicine) extended to mean nonsense.
Edit 2: Green's Dictionary of Slang Online has the noun back to 1957:
With * go bananas* from 1964 and drive bananas from 1975.