I would say it's the same as with other derogatory words, it depend on who is using it and in what context. 

For example the "n-word" is (by some) used in colloquial conversation between African-Americans, but would be considered derogatory used by a white person.

And when in doubt, don't use possibly derogatory words, better safe than sorry.