As a general rule, adjectives are likely to reflect the meaning of the corresponding noun if they're derived via a derivational suffix (-al, -ial, -ous, -ful, -worthy etc) that is still productive. And vice versa if the noun is derived using a productive suffix (-ness).
If the noun isn't actually derived from the related adjective, or derived in a way that isn't very productive, there's probably more chance of one of the words taking on a figurative meaning without the other necessarily following suit. In the case of yellow, I think you could consider that the adjective essentially "came first" and isn't derived from the noun as such; notice that yellowness, derived productively from yellow, can have the related figurative meaning of "cowardliness".