What are the origin and history of the phrases "local color" and "color commentary"? There is a tiny bit in the dictionaries about this use of color to mean 'additional detail and anecdotes' but not enough to really explain how it came to be.
The Online Etymology dictionary says that 'local color' was first attested in 1721 and was originally a technical term in art.(1) An object's local color is its color under daylight, unmodified by shadows, reflections, or the artist's imagination.(2)
I think it is reasonable to assume that the use you have in mind is a product of Victorian orientalism, as some writers gave "realistic" descriptions of the places they visited.