Why do we say that an obscene joke is "off-color"? Is a G-rated joke "on-color"? What color? When and how did this idiomatic expression come from?
The first definition in the OED is specific to diamond mining, where an off-colour diamond is neither pure white or another colour, which makes it of inferior value (there are quotations using the phrase from 1860 - 1968).
The next definition is more general and has two sub-definitions. The first means of a colour that's either darker, lighter, not natural, proper or acceptable (quoted 1873 - 2000). The second is to be slightly unwell, or not up to the mark, or out of order (1876 - 1997).
Finally, the third is what we're after:
The first quotation:
Searching Google Books, I found some antedatings. First for diamond mining:
The Mirror of Literature, Amusement, and Instruction - Volume 5 - Page 15 - John Timbs - 1825:
Next, for a general use (though I'm not sure if this means "unwell" or "of questionable taste"):
Paris in '67: Or, The Great Exposition, Its Side-shows and Excursions - Page 87 - Henry Morford - 1867:
I have no concrete evidence to suggest so, but I feel as though my following reasoning might help you. In the world of comedy there is a certain genre called "black comedy", this genre focuses on making satirical humor based on actual events that one might consider not funny. One example is the movie "Dr. Strangelove" a black comedy that was satire to the cold war. In the movie there is a scene where a man is forced to ride an atomic bomb down to its detonation, he does this on purpose because otherwise it wouldn't detach from its loading dock. The comedic part is that as he rides it he rides it similar to a cowboy and a bull, making fun of the fact that only an American would be so silly as to make fun out of big explosions. But the "black" aspect is the actuality that the American is conveying the death of millions and ensuring "mutually ensured destruction" for both the soviets and the US in nuclear war. Therefore one can assume that when a joke is "off color" it is not pure, it has a tinge of controversial meaning that some might find offensive.
Here is the definition of off color in Chapman & Kipfer, Dictionary of American Slang, Third Edition (1995):
That same reference reports that blue in the sense of "lewd, rude, suggestive" appeared in American English by 1840.
The same sense of blue appears in England, too. Thus, Farmer & Henley, Slang and Its Analogues (1890) has this entry:
And for blue sense 2, Farmer & Henley says this:
From About.com's definition of blue humor (http://comedians.about.com/od/glossary/g/bluehumor.htm):
The alternative term for clean humor seems to have been white at one time. Wentworth & Flexner, Dictionary of American Slang (1960) offers this item under its entry for blue:
So putting all of this together, we have the idea that off color refers to blue, which means crude, lewd, or indecent; but white or brown indicate clean (or boring). I especially like the idea from Farmer & Henley that blue may have gotten its unsavory connotation from a particular edition of smutty French literature.
As no-one has really emphasised the following meaning I feel justified in offering little more than anecdote: off-colour most definitely means "unwell" where I am from (though it is a little archaic). The "colour" in question being the colour of one's skin.
In the case of a joke, I would see it as a synonym for "sick", meaning distasteful. I'm unsure of the connection between taste and wellness, but if you read things like Frankenstein, one might be led to believe the connection between beauty (i.e. notions of taste) and wellness was far more explicit than today.