Adverbs can usually only modify verbs or adjectives. But some words are both adverbs and adjectives.
For example, the word "fast" is both. "The fast man ran fast."
The word "quick" is an adjective, but can be made into an adverb by adding -ly. "The quick man ran quickly." But it would sound wrong to say "The quickly man ran quickly."
Most dictionaries seem to give the word "here" as an adverb, but with special uses. In phrases like "come here" you can see why that is. In phrases like "here we are" or "the man here," there's an argument that actually it's working as an adjective, but an adjective which goes after the noun, which is unusual in English.
Most adverbs though? It would sound strange unless they are also adjectives already.