When we travel around, some people get lost much more easily than others, since they cannot remember directions correctly. Is there any specific word for these kind of people?
Topographical Agnosia or topographical disorientation. The actual term to describe this person is probably "Topographically Agnostic" or "Topographically Disoriented". It is a family of different problems. Some people are quite able to navigate by using GPS but cannot navigate by listening to directions or memory. It is similar to the problem that some people need to write each telephone number on a piece of paper while others can recall many telephone numbers that were only mentioned once.
It is a process of differential sensory (visual and auditory), memory and computational skills. The visual inputs of the brain are processed in a different area from the auditory inputs.
Wikipedia has an excellent discussion of this.
Anterograde Disorientation is this problem but only for location navigation after an event. This is what is typical of a patient with dementia. This is distinct from people who always experienced this process.
There is a cluster of developmental disabilities which include this symptom. The proper word depends on the cause.
Dyscalculia is a disability of number sense. The person is likely to have difficulty remembering directions properly. They can confuse left and right and compass directions, and make very poor estimates of distance traveled.
Dyspraxia is a disability of movement. The person is likely to forget instructions, especially those which involve a sequence of steps. Like dyscalculia, dyspraxia creates confusion of left and right, compass directions, and distance estimates.
There are other disabilities, such as dementia, which can have this effect, but I assume you are not limiting your inquiry to people with a degenerative disease.
I call myself geographically illiterate. I have never been able to remember directions. I could be somewhere 50 times and still not know how to get there. I also have issues with numbers. If smart phones weren't invented, I would always be at home with nobody to talk to. I tell people "I need gps to check my mail and my mailbox is attached to my house". Sometimes I'm surprised I can remember the way to my bedroom from my living room :/