We say “spelling error”, which seems to imply it would be “grammar error” since both spelling and grammar are nouns, whereas grammatical is an adjective, but I can’t help think “grammatical error” is what I would say, and feels more correct. But it bothers me that the two phrases don’t have the same grammatical structure.
Similarly, I don’t think I would ever say “typography error”, but instead “typographical error”.
There is, apparently, “orthographical” as an near-equivalent form for “spelling”.