When we travel around, some people get lost much more easily than others, since they cannot remember directions correctly. Is there any specific word for these kind of people?
There is a cluster of developmental disabilities which include this symptom. The proper word depends on the cause.
Dyscalculia is a disability of number sense. The person is likely to have difficulty remembering directions properly. They can confuse left and right and compass directions, and make very poor estimates of distance traveled.
Dyspraxia is a disability of movement. The person is likely to forget instructions, especially those which involve a sequence of steps. Like dyscalculia, dyspraxia creates confusion of left and right, compass directions, and distance estimates.
There are other disabilities, such as dementia, which can have this effect, but I assume you are not limiting your inquiry to people with a degenerative disease.
Topographical Agnosia or topographical disorientation. The actual term to describe this person is probably "Topographically Agnostic" or "Topographically Disoriented". It is a family of different problems. Some people are quite able to navigate by using GPS but cannot navigate by listening to directions or memory. It is similar to the problem that some people need to write each telephone number on a piece of paper while others can recall many telephone numbers that were only mentioned once.
It is a process of differential sensory (visual and auditory), memory and computational skills. The visual inputs of the brain are processed in a different area from the auditory inputs.
Wikipedia has an excellent discussion of this.
Anterograde Disorientation is this problem but only for location navigation after an event. This is what is typical of a patient with dementia. This is distinct from people who always experienced this process.