They won't be called one's 'enemies' because they are not actively opposed or hostile to one. They won't be one's adversaries, opponents or rivals either since there is no contest, conflict or dispute between. But they are people who are not on good terms with someone for any other imaginable reason. For example, religious intolerance or feelings of general dislike due to differences of social class or standard of living, language, lifestyle etc.
There is a specific word in our native language to match the description that I have given above and I am interested to know if there is an English equivalent of it.
P.S.: Please feel free to edit this topic if it's not clear enough.