This is a matter of the particular language you consider. There is no variation to speak of for fundamental words, for instance, among European languages, words such as nouns and verbs; I mean by that that if the word that refers to a material object is a noun in one language it is also a noun in the other languages, as "messer" is a noun in German, "knife" is a noun in English, "couteau" is a noun in French and "cuchillo" a noun in Spanish, the four words referring to the same thing.
As the nature of the word becomes more difficult to grasp conceptually, which makes the word likely to be associated in various ways with other words, differences begin to be manifest among the various languages. As time goes by a more or less definite usage is confirmed and the word acquires a status which is characteristic of the language in which you find it.
For instance, the word "here" cannot normally be found in attributive position in English, yet there is a tendency, although rather negligible, to consider that it can, as many books show (this here book, this here house). In French, for instance, there is as yet absolutely no possibility to use "here" ("ici") as an adjective, which shows that where the nature of the word is difficult to make out, language decides, so to speak.
However, we should say in familiar terms that "here" is not yet much of an adjective in English. We can understand that grammatically by considering what defines an adjective, and by doing so we will have gained the idea that the mere label "adjective" in a dictionary is not the identification of a hard and fast category; such a word is some of the time less than an ideal adjective. To see that we refer to CoGEL's study of the class of adjectives; it is shown that there exists what is called a gradient between the quintessential adjective and adverb. (The line "7'" is not part of the table; I inserted it as an illustration for further discussion to follow.)
CoGEL Table 7.3
Criteria for establishing adjective classes
(a) attributive use
(b) predicative use after the copula seem
(c) premodification by very
(d) comparison
|
|
(a) |
(b) |
(c) |
(d) |
feature |
class |
[1] |
hungry |
+ |
+ |
+ |
+ |
central |
ADJECTIVES |
[2] |
infinite |
+ |
+ |
- |
- |
" |
" |
[3] |
old |
+ |
- |
+ |
+ |
peripheral |
" |
[4] |
afraid |
? |
+ |
+ |
+ |
" |
" |
[5] |
utter |
+ |
- |
- |
- |
" |
" |
[6] |
asleep |
- |
+ |
- |
- |
" |
" |
[7] |
soon |
- |
- |
+ |
+ |
|
ADVERBS |
[7'] |
here |
-(+) |
- |
- |
- |
|
" |
[8] |
abroad |
- |
- |
- |
- |
|
" |
CoGEL, § 7.4
If we examine Table 7.3, we see that [1] hungry alone satisfies all four criteria; [2] infinite accepts (a) and (b); […] [8] satisfies none of the four criteria.
Criterion (c), acceptance of premodification by very, and criterion (d), the ability to take comparison, have no diagnostic value in distinguishing adjectives from adverbs. Those two features generally coincide for a particular word, and are determined by a semantic feature, gradability, which cuts across word classes. Thus, as we can see in the table, the adverb soon is gradable, just like the central adjective hungry. […] We consider the ability of functioning both attributively and predicatively to be central features of the adjectives.
Words like hungry and infinite, which satisfy both of these criteria (a and b) are called CENTRAL adjectives.
Words like old, afraid, utter, and asleep, which satisfy at least one of these first two criteria (a or b), are called PERIPHERAL adjectives.
The usage shown above concerning "here" must be examined more seriously. We see that this usage coincides with a preceding demonstrative,"this".

"the here book" and "the here house", as well as "a here book" and "a here house" are not found. Thus, if we are to consider "here" according to the scheme of definition in CoGEL we are bound to say that the word does shows perceptibly the nature of an adjective but remains only very marginally an adjective and is still uniquely an adverb.
Essentially the answer to your question (formulated above) is that usage decides, or in other words the particular language.
- Is there a simple concept or sentence that would clarify the difference? I'm guessing "I am" is the most confusing example.
This part of the question is founded on two acceptations of the verb "to be" and is not really relevant to the question of whether "here" is or not an adjective.
1 [V+adv/prep] to be located; to be in a place
2 as copular verb Having the state, quality, identity, nature, role, etc., specified. (lexico)
The second possibility makes for nonsense, only the first has meaning.
- Is it true that this applies to all/most location-related words? What makes location special? or is that just how the language developed?
Location is special in this respect that it does not "intrinsically" characterize something, whereas the adjective does. So, some languages at least developed along this logical line (English, French, German).
- Are there any other classes of words like this? I checked out time-related words, like "I'm early/late", but I think those are adjectives in that usage, is that right?
Yes, there are prepositions that are considered to be prepositions only marginally. In the words of CoGEL, "there are some words which behave in many ways like prepositions, although they have also affinities with other word classes such as verbs or adjectives".
Here are some of those from CoGEL, § 9.8.
bar, barring, excepting, excluding, save
concerning, following, given
In this context, "late" and "early" are not adjectives but adverbs.