Hazel (Old English: hæsl, hæsel) is of Proto-Germanic: hasalaz. It is related to the Dutch hazelaar meaning ‘hazel tree’ and hazelnoot meaning ‘hazelnut’.
Interestingly, Shakespeare was the first to use it in print, in reference to the reddish-brown colour of eyes in his 1592 play: Romeo and Julliet. Presumably, inspired by the colour of ripe hazel nuts produced by the hazel tree. This is seen when Mercutio accuses Benvolio:
"Thou wilt quarrell with a man for cracking Nuts, hauing no reason, but because thou hast hasell eyes".
Presumably before Middle English, hazel was a noun denoting the tree and a "modifier" when referring to the physical aspects of the tree. In this sense it was an evaluative adjective (something that can be measured and compared):
hazel trees (quantitative)
hazel nuts (quantitative)
I'm guessing the colour "hazel" is a descriptive adjective which tell us more about the non-measurable properties of an object, such as the hazelnuts produced from a hazel tree. For example, when a person claims you are eyes are hazel, or hazelnuts are hazel in colour. These are all being used in the subjective sense unless there are qualifiers for what constitutes a colour to be hazel.
So for example in:
Your eyes are hazel
His eyes are hazel
Your hair has a hazelnut colour
Here hazel is a descriptive adjective.
Similarly, when referring to the measurable aspects of the hazel tree, it is being used evaluatively (it is referring to the tree*):
*A hazel tree has measurable, scientific characteristics of what makes a "hazel tree" "hazel". Hazel is the genus of a tree species.
I gathered 5 hazel nuts from the hazel tree
10 hazel nuts fell from 5 hazel trees.
For example you cannot quantify hazel when referring to the colour and to subjective qualities:
? Your eyes are hazel = are they or is it just the lighting?
? There's 100 hazel eyes in the room = 50 people = 50 people in the room have hazel eyes?
? There are 5 hazel coloured pencils in the room = The 5 coloured pencils are hazel?
Through context it is deducible whether hazel is being used evaluatively or descriptively.
Notice the difference between:
That tree is hazel (descriptive adjective)
That tree is a hazel tree. (evaluative adjective)
Whether "hazel" in
"That tree is a hazel tree"
or
"You have hazel eyes"
functions as a modifier or is an adjective, depends on the lexicon (dictionary) you consult.
Essentially, 1) "You have hazel eyes" and 2) "Your eyes are hazel" mean the same to ODO. While true, syntacticly they're not. In 1) hazel is a modifier and in 2) an adjective.
ODO would cite "hazel" as only as a modifier and not an adjective:

Whereas, Cambridge has 2 separate lexical categories for "hazel" (the colour):


I believe this is the same for "hazel" when referring to the tree. For example, you can't say:
? The tree was hazel
for it to definitely mean "The hazel tree"; it has to be affixed to -tree to remove ambiguity:
The tree was a hazel tree
Similar to hazel nut:
? The nut was hazel (to mean hazel nut?)
? The nut that was hazel was nice
(c.f. The hazel nut was nice)
Hazel in "hazel nut" could be a noun functioning as a modifier where only the noun hazel to refer to the type of tree, exists, since there is not a separate lexical category of an adjective to describe "parts of a hazel tree", unlike its colour counterpart.