What is the origin of the word bats so that it came to mean "mad or insane"?
It comes from "bats in the belfry", a figure of speech drawing an analogy between a person's head and a church belfry, and between bats inhabiting a belfry and mental disturbance.
To follow up on my earlier comment under @Robusto's answer, I've been trying to track down the first print use of the "bats in the belfry" idiom, generally agreed to be the origin of the terms bats and batty to mean not quite sane. Etymonline has 1901 and The Phrase Finder has 1900, but antedatings were not difficult to find via Google.
There's three worth highlighting, all from 1899. The first is from Billy Baxter's Letters by William J. Kountz in which the author humorously describes a scene from an opera:
This reference is significant because it can be specifically dated to early 1899 by the preface of the book and because the author claims the book is full of "up-to-date slang."
The second reference is from Elbert Hubbard's Little Journeys to the Homes of Eminent Painters. In a chapter on the French artist Gustave Doré, Hubbard quotes painter James Whistler dismissing Doré with this comment:
This reference is interesting because Hubbard later refers to the phrase as a joke and a bon mot. It's possible that the joke part is in reference to the bat-like wings of devils portrayed in many of Doré's works.
The third reference, from a late 1899 edition of The Child-Study Monthly, actually alludes back to Hubbard's use of the phrase:
This passage not only confirms the phrase's origin in current slang rather than from Hubbard's book, but also links the phrase with another alliterative idiom for insanity, rats in the garret. While I can't find this idiom in any modern phrase dictionary, I found use of it back to the 1841 Charles Dickens-edited Pic-Nic Papers.
And one final curious reference from an 1883(?) treatise on hypnosis, Private Instructions in the Science and Art of Organic Magnetism by Mrs. Chandos Leigh Hunt Wallace. Among her list of humorous suggestions a hypnotist might put to a mesmerized group, there is this:
There may be no connection between this British reference and the development of the idiom in America 15 years later, but the description here of people being made to look crazy while imagining bats in the belfry seemed too coincidental not to mention.
Doug at Etymonline has edited his listing for batty to account for these 1899 references of bats in (one's) belfry.
The usual term I hear is not bats but batty, about which Etymonline says this: