Firstly, for my first-hand reporting of the usage (and perceived offensiveness) I observed of this word, discussing with a small Japanese community in London and Boston. Gaijin is fine in normal conversation, both between non-Japanese people and between Japanese and non-Japanese.
Secondly, looking up some authorities, none of the dictionaries I have at hand or checked online (Merriam-Webster, Oxford, Cambridge) mark it as offensive or derogatory.
Thirdly, I thought about similar words for other communities. You gave the example of nigger, which is extremely offensive (“black” or “African American” being respectful alternatives). Another one is goy, which can be perceived as offensive depending on context and audience (and for which “Gentile” is a safer alternative).