The sentence fails in terms of both syntax and semantics.
Syntactically, without knowing more about the meaning of the sentence than what is in the sentence, it needs to be changed to the following:
✔ What the Europeans refer to as the habit of dreaming (singular) by the Aborigines is (singular) not what the Aborigines themselves believe it (the habit of dreaming: singular) to be.
Per syntax alone, this is now correct. And if it is interpreted as the habit of dreaming, everything is fine.
Based purely on the sentence alone, this is what I believe it has to be changed to; I believe that they is a mistake, and it should really be it.
However, this is stated in the question:
"Them" is referring to dreams.
This throws the previous analysis into chaos. First, them has to be retained:
What the Europeans refer to as the habit of dreaming (singular) by the Aborigines is (singular) not what the Aborigines themselves believe them to be.
But this makes no sense.
In the first place, it would not be natural to assume that them is referring to dreams. The only plural subjects in the sentence are the Europeans and the Aborigines. Since there are no other plural subjects in the sentence, and it wouldn't make sense for them to be referring to one of those groups, it's not clear what it could be referring to; it looks like a mistake.
In essence, these are the possible interpretations, with the pronoun being replaced by a noun. And dreams is only being provided as the third noun because that's what it's been stated to be:
1. What the Europeans refer to as the habit of dreaming (singular) by the Aborigines is (singular) not what the Aborigines themselves believe the Europeans to be.
2. What the Europeans refer to as the habit of dreaming (singular) by the Aborigines is (singular) not what the Aborigines themselves believe the Aborigines to be.
3. What the Europeans refer to as the habit of dreaming (singular) by the Aborigines is (singular) not what the Aborigines themselves believe dreams to be.
None of those sentences are meaningful.
Looking at the third sentence, which is said to be correct, it can be simplified in the following way:
? The habit of dreaming is not what dreams are.
While this is syntactically correct, it's not an understandable sentence—at least not on its own without further context to explain how it can make sense. Adding back in all of the qualifiers of the original sentence doesn't help it to make any more sense.
If you can somehow make it known that them is referring to dreams, then you could say that the sentence is syntactical. (And the fact that it's syntactical would have no bearing on the fact that the habit if dreaming is singular.) However, it still wouldn't be meaningful—at least not without further context that could make it so.
If they really does refer to dreams and not to the habit of dreaming, then here is one possible way of rephrasing the sentence so that it makes sense on every level:
✔ What the Europeans refer to as the habit of dreaming by the Aborigines does not involve what the Aborigines themselves believe to be dreams.
But without additional context beyond the single sentence, I can't say if that would be a valid rephrasing or not.