It's not you. It's... well, it is you, but it is the same for everybody.
Foreign languages are foreign. They have all different words and grammar and when words or syntax are sort of the same, even then the meaning can be annoyingly slightly different. And pronunciation of sounds (accent) is similar.
You can obviously understand those who speak your language natively very well (almost by definition), and its hard to understand foreigners speaking your language because of their accent.
But the same goes in the foreign language. You've been raised hearing an accent of a certain kind (everybody has an accent, they just can't hear their own) and you're good at hearing it. So even if someone with your accent is speaking a foreign language, you an understand them easier than the person who is better, well... perfect, at the language, a native speaker of it, even despite the fact that this foreign speaker is speaking correctly.
It's hard for you to hear correctly and easier to hear in an accent that is closer to your native language.
There is no particular term for this (maybe there's one among language teachers).
Notice that none of this is English specific. If you are a Thai speaker and you are learning Mandarin, you will understand Thai people speaking Chinese more easily than native Chinese speakers speaking Chinese, even if your second language Chinese teacher is a native Chinese speaker.