I would say that it is better analyzed as a noun than as an adjective.
I don't know, but it seems possible to me that it is a compound word, and I'm not sure it even makes sense to assign a part of speech to a component of a compound word, except for in the sense of "is derived from that part of speech" (e.g. should we say that the "black" in "blackbird" is an adjective? or just that is is derived from an adjective?).
"Color" has no obvious adjectival morphology
The word "color" has no particular formal features that mark it as an adjective (for example, it cannot be divided into a stem and an adjectival suffix like -y, -ive, -al).
It makes sense that "color" has no morphological features characteristic to adjectives, because it is derived etymologically from a Latin and French noun, and is used as a noun in present-day English.
The stress pattern of "color spectrum" is consistent with a noun-noun compound analysis
The word seems to behave like a compound word in that for me, the primary stress can only go on the first syllable of "color" (aside from in highly contrived circumstances, like abnormal stress to emphasize the "spectrum" part). In an adjective-noun sequence, I believe it's usually possible to put the primary stress on the noun. That said, the evidence of stress doesn't seem to rule out "color" being an adjective because it seems like some adjective-noun sequences typically have primary stress on the adjective (see the following Linguistics SE question: Stress rules in English adjective-noun combinations).
The word "color" is certainly used as a noun in other contexts, but is not typically used as an adjective in other contexts
As you mention, adjectives often can be used after the intensifier "very" (e.g. "very good") but "*very color" doesn't seem to be valid. (This test doesn't prove that "color" isn't an adjective, because some adjectives cannot be modified by "very" in some contexts: e.g. "*the very electromagnetic spectrum" sounds bad even though "electromagnetic" has adjectival morphology (the suffix -ic) and is presumably an adjective.)
"Color" also doesn't seem to be used as an adjective in predicative position.
To summarize, as far as I can tell, there is little external evidence supporting an analysis of "color" as an adjective.
Another language, German, has noun-noun compounds that express the same meaning
While there is no guarantee that an English construction will parallel the grammar of a synonymous construction/word in another language, I think evidence from other languages can provide some information about what structures are possible.
When I look up "color spectrum" using Linguee, it turns up a number of German translations that appear to be unambiguous noun-noun compounds, not adjective-verb sequences, such as Farbspektrum, Farbenspektrum, Farbskala, Farbpalette. This indicates that in a language that is somewhat closely related to English, a noun-noun compound can be used to express this meaning.
It is easier to analyze the structure of the German examples because noun-noun compounds are consistently written together without spaces in German, unlike in English.