I finally found the word I was trying to remember: ingraphicacy (the inability to understand maps).
It's quoted in this University of Edinburgh thesis:
In 1965, for example, Balchin & Coleman wrote:
It is hoped that the concepts of graphicacy and ingraphicacy will be
taken up and developed by educationists, to mould the vague idea of
visual aids at large into a more integrated goal of education, and to
carry it down into the earliest stages to take its rightful role as
one of the essential underpinnings. (p. 947)
I couldn't find the actual Balchin & Coleman work online, though.
The librarian reference of the original work is:
Balchin, W.G.V. & Coleman, A.M.  Graphicacy should be the fourth ace in the pack. The Times Educational Supplement, November 5th, 947.
NOTE: Strangely, searching for the word in Google returns mostly pages written in Spanish and Portuguese (quoted the word in English) but no use in pages written in English besides the very thesis where it's mentioned.
EDIT: As @Gnomew noted, graphicacy is indeed in the dictionary. I thought it wasn't: