Why do we say that an obscene joke is "off-color"? Is a G-rated joke "on-color"? What color? When and how did this idiomatic expression come from?
The first definition in the OED is specific to diamond mining, where an off-colour diamond is neither pure white or another colour, which makes it of inferior value (there are quotations using the phrase from 1860 - 1968).
The next definition is more general and has two sub-definitions. The first means of a colour that's either darker, lighter, not natural, proper or acceptable (quoted 1873 - 2000). The second is to be slightly unwell, or not up to the mark, or out of order (1876 - 1997).
Finally, the third is what we're after:
Of questionable taste, disreputable; improper, vulgar; spec. (of language, jokes, etc.) slightly indecent or obscene. Cf. dirty adj. 2.
The first quotation:
1875 J. G. Holland Sevenoaks in Scribner's Monthly Mar. 582/1 Everybody invited her, and yet every body, without any definite reason, considered her a little ‘off color’.
Searching Google Books, I found some antedatings. First for diamond mining:
The Mirror of Literature, Amusement, and Instruction - Volume 5 - Page 15 - John Timbs - 1825:
The smallest flaw, or foul (as it is called) greatly diminishes the price of the diamond; and if it be tinged with yellow, brown, &c., a fault characterised by the technical term off colour, its value falls very considerably, and is frequently reduced from a third to one half.
Next, for a general use (though I'm not sure if this means "unwell" or "of questionable taste"):
Paris in '67: Or, The Great Exposition, Its Side-shows and Excursions - Page 87 - Henry Morford - 1867:
... and yet, though I have no doubt that the lady has been a little 'off color,' and though Mazeppa and the French Spy may not be exactly the thing in which we should like our sisters to 'show themselves' — yet, do you know, I am not only in love, ...
Here is the definition of off color in Chapman & Kipfer, Dictionary of American Slang, Third Edition (1995):
off color adj phr by 1875 Somewhat salacious; risqué; =BLUE: a couple of off-color jokes/ Some of his observations were a bit off color.
That same reference reports that blue in the sense of "lewd, rude, suggestive" appeared in American English by 1840.
The same sense of blue appears in England, too. Thus, Farmer & Henley, Slang and Its Analogues (1890) has this entry:
blueness subs. (common) —Indecency. Smutty talk is described as BLUE, sense 2.
Carlyle, Diderot . "The occasional blueness of both [writings] shall not altogether affright us."
And for blue sense 2, Farmer & Henley says this:
Indecent; smutty; obscene. This may be derived from the blue dress of harlots—although Hotten suggests it as coming from the French Bibliotheque Bleu, a series of books of questionable character. Books of an entirely opposite nature are said to be brown or Quakerish.
From About.com's definition of blue humor (http://comedians.about.com/od/glossary/g/bluehumor.htm):
Definition: "Blue" humor involves material that's typically considered more "adult"; it can include swearing or foul language, sexual or scatological (bathroom) humor. Most blue humor can only be heard on cable TV or satellite radio; comics rarely "work blue" on network talk shows (like The Tonight Show), mostly because of network standards. Many comics choose never to work blue, keeping their acts clean and more appropriate for all ages.
The alternative term for clean humor seems to have been white at one time. Wentworth & Flexner, Dictionary of American Slang (1960) offers this item under its entry for blue:
3 Risque; vulgar; suggesting the obscene. 1930: "...Blue gags = jokes in questionable taste." Variety. 1953: "In burlesque [comedian Red] Buttons has a reputation for being, if not lily white, at least no bluer than..." G. Millstein in NY Times Mag., Feb 22.
So putting all of this together, we have the idea that off color refers to blue, which means crude, lewd, or indecent; but white or brown indicate clean (or boring). I especially like the idea from Farmer & Henley that blue may have gotten its unsavory connotation from a particular edition of smutty French literature.
As no-one has really emphasised the following meaning I feel justified in offering little more than anecdote: off-colour most definitely means "unwell" where I am from (though it is a little archaic). The "colour" in question being the colour of one's skin.
In the case of a joke, I would see it as a synonym for "sick", meaning distasteful. I'm unsure of the connection between taste and wellness, but if you read things like Frankenstein, one might be led to believe the connection between beauty (i.e. notions of taste) and wellness was far more explicit than today.
I have no concrete evidence to suggest so, but I feel as though my following reasoning might help you. In the world of comedy there is a certain genre called "black comedy", this genre focuses on making satirical humor based on actual events that one might consider not funny. One example is the movie "Dr. Strangelove" a black comedy that was satire to the cold war. In the movie there is a scene where a man is forced to ride an atomic bomb down to its detonation, he does this on purpose because otherwise it wouldn't detach from its loading dock. The comedic part is that as he rides it he rides it similar to a cowboy and a bull, making fun of the fact that only an American would be so silly as to make fun out of big explosions. But the "black" aspect is the actuality that the American is conveying the death of millions and ensuring "mutually ensured destruction" for both the soviets and the US in nuclear war. Therefore one can assume that when a joke is "off color" it is not pure, it has a tinge of controversial meaning that some might find offensive.